Interface for Laparoscopic Surgeries - Movement Gestures

ABSTRACT

A method of controlling a surgical tool includes capturing, with an endoscope, real-time images of surgical tools in a field of view within the body cavity. Image processing of the images is performed in real time to detect movement of at least a portion of a first one of the surgical tools. The system determines if the detected movement is within one of the plurality of predetermined protocols of input movement and, if it is, a corresponding predetermined output commands is carried out such that a second one of the surgical tools is robotically maneuvered or activated.

This application is a continuation of U.S. application Ser. No.15/322,452, filed Dec. 28, 2016, which is a US National Phase filingunder 35 USC 371 of International (PCT) Patent Application No.PCT/IL2015/050718, filed Jul. 9, 2015, which claims priority from U.S.Provisional No. 62/022,688, filed Jul. 10, 2014. Each of the foregoingis incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention generally pertains to a system and method forproviding an improved interface for laparoscopic surgeries.

BACKGROUND OF THE INVENTION

Laparoscopic surgery is becoming increasingly popular with patientsbecause the scars are smaller and their period of recovery is shorter.Laparoscopic surgery requires special training of the surgeon orgynecologist and the theatre nursing staff. The equipment is oftenexpensive and is not available in all hospitals. During laparoscopicsurgery it is often required to shift the spatial placement of theendoscope in order to present the surgeon with the optimal view.Conventional laparoscopic surgery makes use of either human assistantsthat manually shift the instrumentation or alternatively roboticautomated assistants (such as JP patent No. 06063003).

In laparoscopic surgery, the surgeon performs the operation throughsmall holes using long instruments and observing the internal anatomywith an endoscope camera. The surgeon's performance is largely dependenton the camera position relative to the instruments and on a stable imageshown at the monitor. In general, the surgeon needs a close-up view ofthe area in which he wants to work, however, there are times when anoverview of a large portion of the working area, such as an overall viewof the interior of the abdomen, is desirable.

U.S. patent application US2006/0281971 discloses a method and apparatusfor presenting three-dimensional data to a physician is provided tofacilitate the flexible navigation of an endoscope and surgicalinstruments with respect to anatomical structures. In accordance with afirst embodiment a first set of data corresponding to a threedimensional model of a patient's anatomy is received. Thisthree-dimensional model may be rendered from images taken in CT or MRIscanning, as discussed above In accordance with this embodiment, thismodel is then combined with a second set of data corresponding to a viewobtained from an endoscope. In another embodiment, the view from theillustrative endoscope is displayed as an inset image on the display ofthe three-dimensional image. In yet another embodiment, thethree-dimensional image comprises a graphical representation of at leasta first surgical instrument, such as said endoscope. The surgeon mayselect among various combinations of views and may zoom in or out fromany particular view.

However, U.S. patent application US2006/0281971 does not disclose ameans of controlling the endoscope.

U.S. Pat. No. 6,714,841 discloses an automated camera endoscope in whichthe surgeon is fitted with a head mounted light source that transmitsthe head movements to a sensor, forming an interface that converts themovements to directions for the mechanical movement of the automatedassistant. Alternative automated assistants incorporate a voice operatedinterface, a directional key interface, or other navigationalinterfaces. The above interfaces share the following drawbacks:

-   -   a. A single directional interface that provide limited feedback        to the surgeon    -   b. A cumbersome serial operation for starting and stopping        movement directions that requires the surgeon's constant        attention, preventing the surgeon from keeping the flow of the        surgical procedure.

Research has suggested that these systems divert the surgeons focus fromthe major task at hand. Therefore technologies assisted by magnets andimage processing have been developed to simplify interfacing control.However, these improved technologies still fail to address anothercomplicating interface aspect of laparoscopic surgery, in that they donot allow the surgeon to signal to automated assistants, to humanassistants or to surgical colleagues which instrument his attention isfocused on.

Hence there is still a long felt need for improving the interfacebetween the surgeon, his surgical colleagues or human assistants and anendoscope system, for laparoscopic surgery

SUMMARY OF THE INVENTION

It is an object of the present invention to disclose a system forproviding improved interface for laparoscopic surgeries.

It is another object of the present invention to disclose a maneuveringsystem, comprising: (a) at least one endoscope adapted to real-timeprovide at least one image of a field of view; (b) at least one surgicaltool; (c) at least one maneuvering mechanism in active communicationwith at least one selected from a group consisting of said at least oneendoscope, said at least one surgical tool and any combination thereofsaid maneuvering mechanism is configured to maneuver at least oneselected from a group consisting of said at least one endoscope, said atleast one surgical tool and any combination thereof in at least twodimensions; (d) a computer program which, when executed by a dataprocessor, is in communication with a member of a group consisting ofsaid at least one endoscope, said at least one surgical tool and anycombination thereof said program, when executed by a data processor isconfigured to (i) real-time image process said at least one image; (ii)detect movement of at least a portion of said at least one surgicaltool; wherein, if said detected movement of at least a portion of saidat least one surgical tool is within a predetermined protocol of inputmovement, then at least one of the following is being held true: (a)said endoscope is maneuvered by means of said maneuvering mechanismaccording to a predetermined protocol of output movement; (b) said atleast one surgical tool is maneuvered by means of said maneuveringmechanism according to a predetermined protocol of output movement; (c)said at least one surgical tool is activated; (d) a second surgical toolis maneuvered by means of said maneuvering mechanism according to saidat least one output movement protocol; (e) a second surgical tool isactivated according to said at least one output movement protocol; and,(f) any combination thereof.

It is another object of the present invention to disclose a system formaneuvering a surgical tool, comprising: (a) at least one surgical tool;(b) at least one endoscope; (c) at least one maneuvering mechanism inactive communication with said at least one surgical tool and said atleast one endoscope; said maneuvering mechanism is configured tomaneuver at least one selected from a group consisting of said at leastone surgical tool, said at least one endoscope and any combinationthereof in at least two dimensions; (d) at least one sensor configuredto indicate at least one movement of at least one moving element; saidsensor indicates movement of said moving element if a current 3Dposition or current signal, 3D_(current), is substantially differentfrom a previous 3D position or previous signal of the same,3D_(previous); (e) either a wired or wireless communicable database forconsecutively storing said 3D_(current) and said 3D_(previous) of eachof said moving element; and (f) a data processor comprising a computerprogram in communication with said at least one surgical tool and saidat least one maneuvering mechanism; said program, when executed by saiddata processor is configured to identify if said movement of said movingelement is within a predetermined protocol of input movement; wherein,if said detected movement of said moving element is within saidpredetermined protocol of input movement or said detected position ofsaid moving element is within a predetermined protocol of inputpositions, then at least one of the following is being held true: (a)said endoscope is maneuvered by means of said maneuvering mechanismaccording to a predetermined protocol of output movement; (b) said atleast one surgical tool is maneuvered by means of said maneuveringmechanism according to a predetermined protocol of output movement; and(c) said at least one surgical tool is activated; (d) a second surgicaltool is maneuvered by means of said maneuvering mechanism according tosaid at least one output protocol; (e) a second surgical tool isactivated according to said at least one output protocol; and, (f) anycombination thereof.

It is another object of the present invention to disclose a maneuveringsystem, comprising: (a) at least one endoscope adapted to real-timeprovide at least one image of a field of view; (b) at least one surgicaltool; (c) at least one maneuvering mechanism in active communicationwith at least one selected from a group consisting of said at least oneendoscope, said at least one surgical tool and any combination thereofsaid maneuvering mechanism is configured to maneuver at least oneselected from a group consisting of said at least one endoscope, said atleast one surgical tool and any combination thereof in at least twodimensions; (d) a computer program which, when executed by a dataprocessor, is in communication with a member of a group consisting ofsaid at least one endoscope, said at least one surgical tool and anycombination thereof said program, when executed by a data processor isconfigured to determine, from said image of said field of view, an inputprotocol; wherein, if said input protocol is within a predeterminedinput command, at least one output command is being activated.

It is another object of the present invention to disclose a system formaneuvering a surgical tool, comprising (a) at least one surgical tool;(b) at least one endoscope; (c) at least one maneuvering mechanism inactive communication with said at least one surgical tool and said atleast one endoscope; said maneuvering mechanism is configured tomaneuver said at least one surgical tool in at least two dimensions; (d)at least one sensor configured to indicate a member of a groupconsisting of: movement of at least one moving element, position of atleast one moving element, and any combination thereof; said sensorindicates said position of said moving element via a current 3D positionor current signal, 3D_(current); and said sensor indicates said movementof said moving element if said current 3D position or current signal,3D_(current), is substantially different from at least one previous 3Dposition or previous signal of the same, 3D_(previous); (e) either awired or wireless communicable database for storing said 3D_(current)for each said moving element, and, to indicate movement, consecutivelystoring said 3D_(current) and said 3D_(previous) of each of said movingelement; and (f) a data processor comprising a computer program incommunication with said at least one surgical tool and said at least onemaneuvering mechanism; said program, when executed by said dataprocessor is configured to identify at least one of a group consistingof: said movement of said moving element is within a predetermined inputprotocol; said position of said moving element is within a predeterminedinput protocol and any combination thereof, wherein, if said inputcommand is within a predetermined input protocol, at least one outputcommand is being activated.

It is another object of the present invention to disclose the system asdescribed above, wherein said input command is selected from a groupconsisting of: at least one said surgical tool is maneuvered accordingto an input movement protocol; at least a portion of at least one saidsurgical tool is positioned in a predetermined region of said field ofview; at least a portion of at least one said surgical tool ispositioned less than a predetermined distance from an edge of said fieldof view; at least a portion of at least one said surgical tool isoriented at a predetermined angle in said field of view; at least onesaid surgical tool is activated; at least one said surgical tool isdeactivated; at least one said surgical tool is articulated; at leastone said endoscope is maneuvered according to an input movementprotocol; at least one said endoscope is articulated; at least one saidendoscope is zoomed; at least one said endoscope is activated; at leastone said endoscope is deactivated; at least a portion of a secondendoscope is positioned in a predetermined region of said field of view;at least a portion of a second endoscope is positioned less than apredetermined distance from an edge of said field of view; at least aportion of a second endoscope is oriented at a predetermined angle insaid field of view; a relationship between at least two articles; andany combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said input movement protocol is selected from agroup consisting of: moving said at least one surgical tool parallel tothe X axis; moving said at least one surgical tool parallel to the Yaxis; moving said at least one surgical tool parallel to the Z-axis;rotational movement of said at least one surgical tool around an axisparallel to the X axis; rotational movement of said at least onesurgical tool around an axis parallel to the Y axis; rotational movementof said at least one surgical tool around an axis parallel to the Zaxis; shaking said at least one surgical tool, moving said at least onesurgical tool in at least a portion of a circle, moving said at leastone surgical tool in at least a portion of an oval, moving said at leastone surgical tool in at least a portion of an ellipse, moving said atleast one surgical tool in a straight line, moving said at least onesurgical tool in a zigzag, moving said at least one endoscope parallelto the X axis; moving said at least one endoscope parallel to the Yaxis; moving said at least one endoscope parallel to the Z-axis;rotational movement of said at least one endoscope around an axisparallel to the X axis; rotational movement of said at least oneendoscope around an axis parallel to the Y axis; rotational movement ofsaid at least one endoscope around an axis parallel to the Z axis;shaking said surgical tool, moving said at least one endoscope in atleast a portion of a circle, moving said at least one endoscope in atleast a portion of an oval, moving said at least one endoscope in atleast a portion of an ellipse, moving said at least one endoscope in astraight line, moving said at least one endoscope in a zigzag, and anycombination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said activation is selected from a groupconsisting of: opening said at least one surgical tool, closing said atleast one surgical tool, causing said at least one surgical tool tofunction, stopping said at least one surgical tool from functioning,introducing at least one said surgical tool to the surgical environment,removing at least one said surgical tool from the surgical environment,introducing at least one said endoscope to a surgical environment,removing at least one said endoscope from a surgical environment and anycombination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said output command is selected from a groupconsisting of: said at least one surgical tool is maneuvered accordingto an output movement protocol; at least a portion of at least one saidsurgical tool is repositioned to a predetermined region of said field ofview; at least a portion of at least one said surgical tool is orientedat a predetermined angle in said field of view; said at least onesurgical tool is activated; at least one said surgical tool isdeactivated; in at least one said surgical tool, at least one of a groupconsisting of an articulation angle, an articulation length and anycombination thereof is altered; at least one said surgical tool istagged; at least one said surgical tool is tracked; a second surgicaltool is maneuvered according to said at least one output protocol; asecond surgical tool is activated; a second surgical tool isdeactivated; in a second surgical tool, at least one of a groupconsisting of an articulation angle, an articulation length and anycombination thereof is altered; at least one said endoscope ismaneuvered according to an output movement protocol; in at least onesaid endoscope, at least one of a group consisting of an articulationangle, an articulation length and any combination thereof is altered; atleast one said endoscope is zoomed; at least one said endoscope isactivated; at least one said endoscope is deactivated; at least aportion of a second endoscope is positioned in a predetermined region ofsaid field of view; at least a portion of of a second endoscope ispositioned less than a predetermined distance from an edge of said fieldof view; a second endoscope is tagged; a second endoscope is tracked; atleast a portion of an organ is tagged; at least a portion of a bodystructure is tagged; a relationship between at least two articles; andany combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said output movement protocol is selected froma group consisting of: moving said at least one surgical tool parallelto the X axis; moving said at least one surgical tool parallel to the Yaxis; moving said at least one surgical tool parallel to the Z-axis;rotational movement of said at least one surgical tool around an axisparallel to the X axis; rotational movement of said at least onesurgical tool around an axis parallel to the Y axis; rotational movementof said at least one surgical tool around an axis parallel to the Zaxis; shaking said at least one surgical tool, moving said at least onesurgical tool in at least a portion of a circle, moving said at leastone surgical tool in at least a portion of an oval, moving said at leastone surgical tool in at least a portion of an ellipse, moving said atleast one surgical tool in a straight line, moving said at least onesurgical tool in a zigzag, moving said endoscope parallel to the X axis;moving said endoscope parallel to the Y axis; moving said endoscopeparallel to the Z-axis; rotational movement of said endoscope around anaxis parallel to the X axis; rotational movement of said endoscopearound an axis parallel to the Y axis; rotational movement of saidendoscope around an axis parallel to the Z axis; shaking said at leastone surgical tool, moving said endoscope in at least a portion of acircle, moving said endoscope in at least a portion of an oval, movingsaid endoscope in at least a portion of an ellipse, moving saidendoscope in a straight line, moving said endoscope in a zigzag, anallowed movement, a restricted movement and any combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said output protocol is selected from a groupconsisting of: said at least one endoscope is maneuvered such that atleast one said surgical tool remains within said field of view, said atleast one endoscope is maneuvered such that at least one said surgicaltool is at a center of said field of view, said at least one endoscopeis maneuvered such that at least one said surgical tool is more than apredetermined distance from said edge of said field of view, said atleast one endoscope is maneuvered such that at least one said surgicaltool is between said edge of said field of view and said predetermineddistance from aid edge of said field of view, zoom of said at least oneendoscope is altered until said at least one surgical tool at a distancefrom said edge of said field of view than said predetermined distancefrom said edge of said field of view and any combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said activation of said at least one surgicaltool is selected from a group consisting of: opening said at least onesurgical tool, closing said at least one surgical tool, causing said atleast one surgical tool to said surgical, stopping said at least onesurgical tool from functioning and any combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said relationship is selected from a groupconsisting of: a predetermined distance between said articles, apredetermined angle between said articles, at least one said article ina predetermined orientation with respect to at least one other saidarticle, a predetermined difference in speed between at least twoarticles, a predetermined difference in velocity between at least twoarticles, and any combination thereof.

It is another object of the present invention to disclose the system asdescribed above, wherein said article is selected from a groupconsisting of: a surgical tool, an endoscope, at least a portion of atool, at least a portion of an endoscope, at least a portion of a body,at least a portion of an organ, at least a portion of a tissue, at leasta portion of an object and any combination thereof, where tissue refersto a structure in the body including, but not limited to, a membrane, aligament, fat, mesentery, a blood vessel, a nerve, bone, cartilage, atumor, a cyst and any combination thereof and an object can include aswab, suture thread, a towel, a sponge, a knife blade, a scalpel blade,a pin, a safety pin, a tip, tube, an adapter, a guide such as a cuttingguide, a measurement device and any combination thereof

It is another object of the present invention to disclose a system forproviding at least one augmented reality image, comprising: (a) at leastone tool; (b) at least one camera located in said tool, configured toreal-time provide at least one image of a field of view of said tool;and (c) a computer program which, when executed by data processor, isconfigured to real time generate a display image by at least one of agroup consisting of: (i) generating at least one virtual marker at atleast one predetermined position within at least a portion of saidimage; (ii) rendered superimposing of at least a portion of said imageand at least a portion of a second imaging modality image; wherein saidsuperposition or said marking in said image is unaffectable by changesin said display image.

It is another object of the present invention to disclose a method formaneuvering an endoscope, comprising steps of: (a) providing amaneuvering system, comprising: (i) at least one endoscope adapted toreal-time provide at least one image of a field of view of the same;(ii) at least one surgical tool; (iii) at least one maneuveringmechanism in active communication with a member of a group consisting ofsaid at least one endoscope, said at least one surgical tool and anycombination thereof said maneuvering mechanism is configured to maneuverin at least two dimensions a member of a group consisting of said atleast one endoscope, said at least one surgical tool and any combinationthereof (iv) a computer program which, when executed by a dataprocessor, is in communication with a member of a group consisting ofsaid at least one endoscope, said at least one surgical tool and anycombination thereof said program, when executed by a data processor isconfigured to (i) real-time image process at least one image; (ii)detect movement of at least a portion of said at least one surgicaltool; (b) real-time providing said at least one image of said field ofview; (c) real-time image processing said at least one image; (d)real-time detecting movement of said at least a portion of said at leastone surgical tool; thereby, if said detected movement of said at leastone surgical tool is within a predetermined protocol of input movement,executing at least one of the following steps: (a) maneuvering saidendoscope by means of said maneuvering mechanism according to apredetermined protocol of output movement, (b) maneuvering said surgicaltool by means of said maneuvering mechanism according to a predeterminedprotocol of output movement; (c) activating said at least one surgicaltool; (d) maneuvering a second surgical tool by means of saidmaneuvering mechanism according to said at least one output protocol;(e) activating a second surgical tool according to said at least oneoutput protocol; and (f) any combination thereof.

It is another object of the present invention to disclose a method formaneuvering of a surgical tool, comprising steps of: (a) providing asurgical tool maneuvering system, comprising: (i) at least one surgicaltool; (ii) at least one endoscope; (iii) at least one maneuveringmechanism in active communication with said at least one surgical tooland said at least one endoscope; said maneuvering mechanism isconfigured to maneuver at least one selected from a group consisting ofsaid tool, said at least one endoscope and any combination thereof in atleast two dimensions; (iv) at least one sensor configured to indicate atleast one movement of at least one moving element; said sensor indicatesmovement of said moving element if a current 3D position or currentsignal, 3D_(current), is substantially different from a previous 3Dposition or previous signal of the same, 3D_(previous); (v) either awired or wireless communicable database for consecutively storing said3D_(current) and said 3D_(previous) of each of said moving element; and(vi) a data processor comprising a computer program in communicationwith said at least one surgical tool and said at least one maneuveringmechanism; said program, when executed by said data processor isconfigured to identify if said movement of said moving element is withina predetermined protocol of input movement; (b) indicating said at leastone movement of said at least one moving element by means of saidsensor; (c) storing said 3D_(current) and said 3D_(previous) of each ofsaid moving element upon indication of said movement from said sensor;and (d) identifying if said movement of said moving element is within apredetermined protocol of input movement thereby, if said detectedmovement of said moving element is within said predetermined protocol ofinput movement movement or said detected position of said moving elementis within a predetermined protocol of input positions, maneuvering, bymeans of said maneuvering mechanism, said member of a group consistingof at least one surgical tool, said at least one endoscope and anycombination thereof according to a predetermined protocol of outputmovement, additionally comprising at least one of the following steps:(a) maneuvering said endoscope by means of said maneuvering mechanismaccording to a predetermined protocol of output movement; (b)maneuvering said at least one surgical tool by means of said maneuveringmechanism according to a predetermined protocol of output movement; (c)activating said at least one surgical tool; (d) maneuvering a secondsurgical tool is maneuvered by means of said maneuvering mechanismaccording to said at least one output protocol; (e) activating a secondsurgical tool according to said at least one output protocol and, (f)any combination thereof.

It is another object of the present invention to disclose a method formaneuvering an endoscope, comprising steps of: providing a maneuveringsystem, comprising: (i) at least one endoscope adapted to real-timeprovide at least one image of a field of view; (ii) at least onesurgical tool; (iii) at least one maneuvering mechanism in activecommunication with at least one selected from a group consisting of saidat least one endoscope, said at least one surgical tool and anycombination thereof; said maneuvering mechanism is configured tomaneuver at least one selected from a group consisting of said at leastone endoscope, said at least one surgical tool and any combinationthereof; in at least two dimensions; (iv) a computer program which, whenexecuted by a data processor, is in communication with a member of agroup consisting of said at least one endoscope, said at least onesurgical tool and any combination thereof; said program, when executedby a data processor is configured to identify, from said image of saidfield of view, an input command; (b) real-time providing said at leastone image of said field of view; (c) real-time identifying said inputcommand; thereby, if said input command is within a predeterminedprotocol of input movement, activating at least one output protocol.

It is another object of the present invention to disclose a method formaneuvering an endoscope, comprising steps of: (a) providing amaneuvering system, comprising (i) at least one surgical tool; (ii) atleast one endoscope; (iii) at least one maneuvering mechanism in activecommunication with said at least one surgical tool and said at least oneendoscope; said maneuvering mechanism is configured to maneuver said atleast one surgical tool in at least two dimensions; (iv) at least onesensor configured to indicate at least one of position of at least onemoving element, movement of at least one moving element, and anycombination thereof; (v) said sensor indicates said position of saidmoving element from a current 3D position or a current signal,3D_(current), and indicates said movement of said moving element if3D_(current) is substantially different from a previous 3D position or aprevious signal of the same, 3D_(previous); (vi) either a wired orwireless communicable database for consecutively storing said3D_(current) and said 3D_(previous) of each of said moving element; and(vii) a data processor comprising a computer program in communicationwith said at least one surgical tool and said at least one maneuveringmechanism; said program, when executed by said data processor isconfigured to identify at least one of a group consisting of: saidmovement of said moving element is within a predetermined inputprotocol; said position of said moving element is within a predeterminedinput protocol and any combination thereof; (b) indicating said at leastone of position of said at least one moving element, movement of said atleast one moving element and any combination thereof by means of saidsensor; and (c) identifying if said position of said moving element orsaid movement of said moving element is within a predetermined inputprotocol; thereby, if said input command is within a predetermined inputprotocol, activating at least one output command.

It is another object of the present invention to disclose a method forgenerating at least one augmented reality image, comprising steps of:(a) providing a system for generating at least one augmented realityimage, comprising: (i) at least one tool; (ii) at least one cameralocated in said tool, configured to real-time provide at least one imageof a field of view of said tool; and (iii) a computer program which,when executed by data processor, is configured to real time generate adisplay image by a member of a group consisting of: (a) generating atleast one virtual marker at at least one predetermined position withinat least a portion of said image; (b) rendered superimposing of at leasta portion of said image and at least a portion of a second imagingmodality image and (c) any combination thereof; (b) generating at leastone said image of said field of view; and (c) executing said computerprogram, thereby carrying out a step selected from a group consistingof: (a) generating a virtual marker at at least one predeterminedposition within at least a portion of said image, (b) generating atleast one rendered superposition of at least a portion of said image andat least a portion of a second imaging modality and (c) any combinationthereof wherein said superposition or said marking in said image isunaffectable by changes in said display image.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it may beimplemented in practice, a plurality of embodiments will now bedescribed, by way of non-limiting example only, with reference to theaccompanying drawings, in which

FIG. 1A-D illustrates an embodiment of a collision avoidance function;

FIG. 2A-D illustrates an embodiment of a no-fly zone function;

FIG. 3A-D illustrates an embodiment of a preferred volume zone function;

FIG. 4 illustrates an embodiment of an organ detection function;

FIG. 5 illustrates an embodiment of a tool detection function;

FIG. 6A-B illustrates an embodiment of a movement detection function;

FIG. 7A-D illustrates an embodiment of a prediction function;

FIG. 8 illustrates an embodiment of a right tool function;

FIG. 9A-B illustrates an embodiment of a field of view function;

FIG. 10 illustrates an embodiment of a tagged tool function;

FIG. 11A-C illustrates an embodiment of a proximity function;

FIG. 12A-B illustrates an embodiment of an operator input function;

FIG. 13A-D illustrates an embodiment of a constant field of view rule;

FIG. 14 illustrates an embodiment of a change of speed rule;

FIGS. 15A-B and 16A-B illustrate embodiments of tool gesture inputmovement protocols;

FIGS. 17A-C, 18A-C and 19A-C illustrate embodiments of hand gestureinput movement protocols;

FIG. 20A-C illustrates an embodiment of an eye gesture input movementprotocol;

FIG. 21A-B illustrates an embodiment of a location-based input protocol;

FIG. 22A-B illustrates an embodiment of tagging;

FIG. 23A-B illustrates an embodiment of an action;

FIG. 24A-B illustrates an embodiment of keeping a tool in the field ofview; and

FIG. 25 illustrates an embodiment of a relationship between tools.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description is provided, alongside all chapters of thepresent invention, so as to enable any person skilled in the art to makeuse of said invention and sets forth the best modes contemplated by theinventor of carrying out this invention. Various modifications, however,will remain apparent to those skilled in the art, since the genericprinciples of the present invention have been defined specifically toprovide a means and method for providing augmented reality endoscopicimage.

The term ‘camera’ hereinafter refers to an image acquiring element.Examples of a camera include, but are not limited to, a CCD array and anelectromagnetic system such as a TV camera.

The term ‘endoscope distal end’ hereinafter refers to the end of theendoscope that is inside the patient. The camera is attached to theother side of the endoscope, outside of the patient's abdomen.

The term ‘field of view’ (FOV) hereinafter refers to the scene visibleto the camera.

The term ‘display view’ hereinafter refers to the scene displayable toan operator.

The term ‘structured light’ hereinafter refers to a method of producing3D images using a single 2D camera. In the structured light method, theobject is illuminated by a set of rays of light, each ray illuminating aspot on the object from a known position and a known direction, and eachray emitted at a known time. For each known time, a 2D camera image iscreated from light reflected from the spots created from rays existingat that time. Initially, a known calibration object is illuminated. Fromthe known shape, size and position of the calibration object and fromthe locations in the camera images of the reflected light, mathematicalmatrices can be calculated. These matrices enable calculation of the 3Dlocation of the surface of an unknown object, when the unknown object isilluminated by the same set of rays as illuminated the calibrationobject.

The term ‘virtual marker’ hereinafter refers to a computer-generatedmark, label or other identifier attached to a point or region on thedisplay image. A virtual marker has no physical existence, unlike a tag,wire, or chemical such as luminescent paint physically associated with aportion of the patient.

The terms ‘tool’ and ‘surgical tool’ refer herein to any object usablein a medical procedure. A surgical tools can be, but is not limited to,a scalpel, a grasper, a tweezers, a laparoscope, an endoscope, a trocar,a canula, a swab, a tube, a saw, a chisel, a pair of scissors, a pair ofshears, a knife, a drill, a rasp, a calipers, a cautery, a curette, adilator, a Pinzette, a forceps, a clamp, a hook, a lancet, a luxator, acatheter, a holder, an elevator, a probe, a retractor, a spreader, aspatula, a speculum, a needle, a mesh, a spoon, a stapler, a suture, anda tissue expander.

The term ‘toggle’ refers hereinafter to switching between one taggedsurgical tool to another.

The term ‘surgical environment’ refers hereinafter to any anatomicalpart within the human body which may be in surrounding a surgicalinstrument. The environment may comprise: organs, body portions, wallsof organs, arteries, veins, nerves, a region of interest, or any otheranatomical part of the human body.

The term ‘region of interest’ refers hereinafter to any region withinthe human body which may be of interest to the operator of the system ofthe present invention. The region of interest may be, for example, anorgan to be operated on, a restricted area to which approach of asurgical instrument is restricted, a surgical instrument, or any otherregion within the human body.

The term ‘spatial position’ refers hereinafter to a predeterminedspatial location and/or orientation of an object (e.g., the spatiallocation of the endoscope, the angular orientation of the endoscope, andany combination thereof).

The term ‘prohibited area’ refers hereinafter to a predetermined area towhich a surgical tool (e.g., an endoscope) is prohibited to be spatiallypositioned in.

The term ‘preferred area’ refers hereinafter to predetermined area towhich a surgical tool (e.g., an endoscope) is allowed and/or preferredto be spatially positioned in.

The term ‘automated assistant’ refers hereinafter to any mechanicaldevice (including but not limited to a robotic device) that can maneuverand control the position of a surgical or endoscopic instrument, andthat can in addition be configured to receive commands from a remotesource.

The term ‘tool’, ‘surgical tool’ or ‘surgical instrument’ refershereinafter to any instrument or device introducible into the humanbody. The term may refer to any location on the tool. For example it canrefer to the tip of the same, the body of the same and any combinationthereof. It should be further pointed that the following description mayrefer to a surgical tool/instrument as an endoscope.

The term ‘provide’ refers hereinafter to any process (visual, tactile,or auditory) by which an instrument, computer, controller, or any othermechanical or electronic device can report the results of a calculationor other operation to a human operator.

The term ‘automatic’ or ‘automatically’ refers to any process thatproceeds without the necessity of direct intervention or action on thepart of a human being.

The term ‘allowed movement’ refers hereinafter to any movement of asurgical tool which is permitted according to a predetermined set ofrules.

The term ‘restricted movement’ refers hereinafter to any movement of asurgical tool which is forbidden according to a predetermined set ofrules. For example, one rule, according to the present invention,provides a preferred volume zone rule which defines a favored zonewithin the surgical environment. Thus, according to the presentinvention an allowed movement of a surgical tool or the endoscope is amovement which maintains the surgical tool within the favored zone; anda restricted movement of a surgical tool is a movement which extracts(or moves) the surgical tool outside the favored zone.

The term ‘time step’ refers hereinafter to the working time of thesystem. At each time step, the system receives data from sensors andcommands from operators and processes the data and commands and executesactions. The time step size is the elapsed time between time steps.

The term ‘electroencephalographic pattern’ or ‘brain wave’ refershereinafter to a pattern of electrical impulses of a living brain.Electroencephalographic patterns can be indicative of movement of aportion of the body (the brain is sending a signal to control movement),of intent to move a portion of the body (the brain intends to or is ispreparing to send a movement control signal), of an emotional state(worry, fear, pleasure, etc) and any combination thereof. The bodyportion to be moved can include a limb or portion thereof, an eye, themouth, throat or vocal cords (speech), the torso or neck, or any othermovable portion of the body.

Terms in the singular, such as “a tool” or “an endoscope” can refer tomultiple items unless it is clearly stated that there is only one of anobject. For example, a reference to “a tool” covers any number of tools.

Laparoscopic surgery, also called minimally invasive surgery (MIS), is asurgical technique in which operations in the abdomen are performedthrough small incisions (usually 0.5-1.5 cm) as compared to largerincisions of traditional surgical procedures. The key element inlaparoscopic surgery is the use of an endoscope, which is a deviceconfigured for viewing the scene within the body. Either an imagingdevice is placed at the end of the endoscope, or a rod lens system orfiber optic bundle is used to direct the image to the proximal end ofthe endoscope. Also attached is a light source to illuminate theoperative field, inserted through a 5 mm or 10 mm cannula or trocar toenable viewing of the operative field.

The abdomen is usually injected with carbon dioxide gas to create aworking and viewing space. The abdomen is essentially blown up like aballoon (insufflated), elevating the abdominal wall above the internalorgans like a dome. Within this space, various medical procedures can becarried out.

In many cases, other information is available to the operator, such asimages from other imaging modalities, such as MRI images or CT scanimages. In some cases, it can be desirable to label or tag items in thefield of view, such as, for non-limiting example, tools or organs orregions of tissue to be removed.

The device disclosed herein provides a system for maneuvering anendoscope in at least 2 dimensions, where the system is configured todetermine input commands, typically of at least one tool in the field ofview of an endoscope or of at least one moving element within range of asensor.

Once an input command is determined, the system activates apredetermined associated output command. The association between inputcommand and output command can be arbitrary (such as, for non-limitingexample, shaking a tool to zoom an endoscope) or can be related (suchas, for non-limiting example, closing a hand to close a grasper).

Input commands, as described hereinbelow, can comprise (a) predeterminedinput movement protocols for one or more tools or moving elements, (b)predetermined positions of one or more tools or moving elements, (c)predetermined actions of one or more tools or moving elements, (d)predetermined repositioning of one or more tools or moving elements, andany combination thereof Δny combination of the above can comprise aninput command.

Predetermined input movement protocols of a tool or other moving elementcan include, but are not limited to: shaking the tool or other movingelement, moving a tool or other moving element in at least a portion ofa circle, moving a tool or other moving element in at least a portion ofan oval, moving a tool or other moving element in at least a portion ofan ellipse, moving a tool or other moving element in a straight line,moving a tool or other moving element in a zigzag, rotating a tool orother moving element in a predetermined manner, translating a tool orother moving element in a predetermined manner, and any combinationthereof.

Predetermined input positions of a tool or other moving element caninclude but are not limited to positioning the tool or other movingelement at a predetermined location within a field of view, orientingthe tool or other moving element at a predetermined angle within a fieldof view, and any combination thereof.

The predetermined location in the field of view can be an edge of thefield of view or a predetermined region within the field of view.

Predetermined actions of a tool can include, but are not limited tooperating a tool, activating a tool, articulating a tool, articulatingan endoscope, zooming an endoscope, and any combination thereof.

Repositioning a tool typically refers to moving a tool from one positionor orientation to at least one second position or orientation, wherethere is a predetermined difference between the first position and/ororientation and the second position and/or orientation. For non-limitingexample, repositioning a cautery from the edge of a field of view to thecenter of the field of view can be associated with a command to turn thecautery on; to turn it off, reposition it from the center of the fieldof view to the edge.

Output movement commands can include, but are not limited to, apredetermined output movement protocol for at least one tool or at leastone endoscope, (b) a predetermined position of at least one tool or atleast one endoscope, (c) a predetermined action of of at least one toolor at least one endoscope, (d) a predetermined repositioning of of atleast one tool or at least one endoscope, and any combination thereof.

Predetermined output movement protocols for at least one surgical toolor endoscope can include: tracking at least one surgical tool,repositioning at least one surgical tool, repositioning at least oneendoscope, zooming at least one endoscope, moving at least one surgicaltool in at least a portion of a circle, moving at least one surgicaltool in at least a portion of an oval, moving at least one surgical toolin at least a portion of an ellipse, moving at least one surgical toolin a straight line, moving at least one surgical tool in a zigzag,rotating at least one surgical tool in a predetermined manner,translating at least one surgical tool in a predetermined manner,tracking at least one endoscope, repositioning at least one endoscope,repositioning at least one endoscope, zooming at least one endoscope,moving at least one endoscope in at least a portion of a circle, movingat least one endoscope in at least a portion of an oval, moving at leastone endoscope in at least a portion of an ellipse, moving at least oneendoscope in a straight line, moving at least one endoscope in a zigzag,rotating at least one endoscope in a predetermined manner, translatingat least one endoscope in a predetermined manner, and any combinationthereof.

Output actions include, but are not limited to: tagging at least onesurgical tool, tagging at least one object in a field of view,activating at least one surgical tool, deactivating at least onesurgical tool, articulating at least one surgical tool, tagging at leastone endoscope, activating at least one endoscope, deactivating at leastone endoscope, articulating at least one endoscope, zooming at least oneendoscope, and any combination thereof.

Activation of a tool can include, but is not limited to: opening a tool,closing a tool, causing a tool to function (non-limiting examplesinclude heating a cautery or ablator, starting a drill rotating, andstarting flow of fluid via a tube or canula), and stopping a tool fromfunctioning.

In some embodiments, the endoscope is configured to provide, in realtime, at least one image of its field of view. In such embodiments, theat least one image is analyzed in real time to detect movement of atleast one surgical tool or endoscope in the field of view, therebydetecting movement protocols and identifying input commands.

Other non-limiting examples of means of detecting movement protocols inorder to identify input commands can be: detecting movement protocolsvia at least one sensor in range of at least a portion of at least onetool; detecting movement protocols via movement commands for amaneuvering system; detecting movement protocols via a at least onesensor in range of at least a portion of a maneuvering system;identifying movement protocols via a at least one sensor in range of abody part, via sound, via a sensor configured to detect at least onebrain signal such as an encephalographic pattern, via a sensorconfigured to detect at least one muscular signal, such as an electricor magnetic signal associated with muscular movement, via contact with aprepared surface and any combination thereof.

A non-limiting example of an input movement protocol is shaking of atool tip, and a non-limiting example of an associated output movementprotocol is tracking a tool, either the shaken tool or another tool,where “tracking” refers to maneuvering an endoscope so as to retain thetip of the tool in the center of the field of view of the endoscope.Other input and output protocols will be discussed below

In some embodiments, the system can maneuver the endoscope in twodimensions; in others, it can maneuver the endoscope in at least 3dimensions. The dimensions of maneuvering can involve a combination oflinear and rotational movement, including zooming. Maneuvering caninclude both physical maneuvering and virtual maneuvering. Virtualmaneuvering occurs when at least one aspect of the image is alteredunder processor control, rather than by physically moving the endoscope.For example, instead of moving an endoscope closer to an object at thecenter of the field of view, the processor can execute a virtual zoom,where the processor manipulates a portion of the image so that thedisplay comprises a magnified image of the center of the field of view.

The moving element can be at least a portion of at surgical tool,movement of the distal end of a surgical tool, movement of at least aportion of the body of at least one operator, intended movement of atleast a portion of the body of at least one operator, a brain signalfrom at least one operator, a sound signal and any combination thereof.

Input movement protocols can comprise movement of an object such as aportion of a body of an operator. Non-limiting examples of inputmovement protocols and exemplary associated output commands include:opening or closing a hand or fingers to command opening or closing agrasper, a pair of scissors or any other openable tool; bending a handor finger to command a change in the articulation of an endoscope orother articulating tool; and making a first to command that at least aportion of a tool be fixed in a predetermined position, such as itspresent position. Many other input movement protocols comprisingmovement of a portion of the body will be obvious.

Non-limiting examples of brain signals include a brain signal indicativeof an order to move a portion of the body (e.g., “open the hand”), abrain signal indicative of intention to move a portion of the body(e.g., “the next step will be opening the hand in order to release thegrasper from the tissue”), a brain signal indicative of a futuredevelopment (e.g., “stop that tool!”; “zoom the endoscope outward”).

A moving element has moved if its current 3D position, 3D_(current), issubstantially different from a previous 3D position, 3D_(previous); atleast one such movement determines a movement protocol. A computerprogram (as executed by a processor) can identify at least one inputmovement protocol and, either alone or in conjunction with a member of agroup consisting of: an input position, an input reposition an inputaction and any combination thereof, can determine an input command andits associated output command and can, based on the output command,instruct the maneuvering system to execute the desired output movementprotocol.

Input movement protocols are typically arbitrary, predefined movements,although they need not be. A non-limiting example of an arbitrary inputmovement protocol is a clockwise circle of a tool tip to identify aninput command for an inward zoom; the associated output movementprotocol can comprise zooming the endoscope to increase magnification ina portion of the field of view. A non-limiting example of anon-arbitrary movement protocol is movement that would bring a tool tipinto dangerously close proximity to an organ; the output movementprotocol can comprise reducing the speed at which the tool tip moves,stopping movement of the tool tip, changing the direction of movement ofthe tool tip and any combination thereof.

Other input movement protocols include, but are not limited to,introducing a tool to the surgical environment, removing a tool from thesurgical environment, and any combination thereof.

A non-limiting example of closing a tool is closing a grasper to retaina swab in position; an input protocol can be opening of the hand; theoutput protocol can be opening the grasper and releasing the swab fromthe grasper. In another example, an input movement protocol ofseparating the hands indicates that an operator is going to work deeperin the tissue with the resulting output movement protocol of movingretractors to further open an incision. Similarly, an input movementprotocol of bringing the hands together can induce an output movementprotocol of relaxing retractors so as to allow an incision to at leastpartially close.

In another non-limiting example, the input protocol comprises the inputaction of activation of a tool (such as, but not limited to, closing agrasper) with an associated output protocol of zooming the endoscope sothat the image of tissue in the neighborhood of the grasper ismagnified. A related input protocol can be opening of the grasper, withan associated output protocol of zooming outward to give an overview ofthe region.

A non-limiting example of an input command comprising a position in afield of view or in a display image is an input command to keep the tipof a tracked tool within a field of view. The output command wouldcomprise a “grey area” extending from the edge of the field of view to apredetermined distance from the field of view. Entry of the tool tipinto the grey area activates an output protocol whereby the endoscope ismaneuvered to keep the tracked tool tip within the field of view. Insome variants, the output protocol comprises maneuvering the endoscopeto put the tracked tool tip in the center of the field of view, in othervariants, the output protocol comprises maneuvering the endoscope toplace the tracked tool tip slightly more than the predetermined distancefrom the edge, in yet other variants, the output protocol comprisesmaneuvering the endoscope so as to keep the tracked tool tip in the greyarea. In yet other embodiments, the zoom of the endoscope is altereduntil the tracked tool is outside the grey area. Other variants combineone or more of the above.

Another non-limiting example of an input command comprising a positionon a display is a display where placing a tool in a predetermined regionof the display activates or deactivates a predetermined output protocol.For instance, moving a tool tip into a region which is at the bottom ofthe display and just to the left of the center results in an outputprotocol of zooming inward (magnifying a portion of the image). Movingthe tool tip into a region which is at the bottom of the display andjust to the right of the center results in an output protocol of zoomingoutward (demagnifying the image). Regions can include, but are notlimited to, a zooming region, a tool tagging region, a region foractivating a tagged tool, a region for deactivating a tagged tool, aregion rotating the image, a region for moving the center of the fieldof view, a region for saving or storing an image, a region of augmentingan image, a region comprising an image to be used for an augmentation, aregion for moving a tagged tool, and any combination thereof.

In some embodiments, a predetermined input command can result in anoutput protocol in which an object is tagged, where the tagged object isidentified during the input protocol. For non-limiting example assume aninput protocol comprising shaking a tool. The output protocol cancomprise: (a) identifying the shaken tool as the tagged object, (b)transferring the tagging to a next tool, (c) a first shake transferstagging to a next tool, with a subsequent shake transferring the taggingto a next-but-one tool, and so on; (d) tagging an object, where theidentity of the object determined by a portion of the tool (such as thetip of the tool) being, in a display image, over the object; (e) taggingan object by placing a portion of the tool over a predetermined locationon a display such as, for non-limiting example, tagging a liver byplacing the tip of the shaken tool over an icon of a liver at the edgeof the display. Other variants will be obvious to one skilled in theart.

In another embodiment of an input protocol resulting in an outputprotocol in which an object is tagged. the input protocol can comprise apointing finger, associated with an output protocol of tagging theobject pointed to in a display image. For non-limiting example, anoperator points to the liver in a display. The liver is then tagged, andthe endoscope moves so that the liver is centered in the field of viewand in the display image. In some variants, the endoscope will zoom sothat the liver is entirely within the field of view, with the edges ofthe liver at or close to the edges of the field of view.

A non-limiting example of an input command comprising more than oneinput movement protocol is exemplified by an input command to activateclosing a grasper. The input command comprises two input movementprotocols—a first input movement protocols of pointing a finger and asecond input movement protocol of closing a hand. The associated outputcommand will comprise an output action of tagging the grasper and asecond output action of closing the grasper.

A protocol, either an input protocol or an output protocol, can comprisea predetermined interaction between at least two articles, where aninteraction involves a relationship between the articles. An article canbe selected from a group consisting of at least a portion of a tool, atleast a portion of an endoscope, at least a portion of a body, at leasta portion of an organ, at least a portion of a tissue, at least aportion of an object and any combination thereof, where tissue refers toa structure in the body including, but not limited to, a membrane, aligament, fat, mesentery, a blood vessel, a nerve, bone, cartilage, atumor, a cyst and any combination thereof and an object can include aswab, suture thread, a towel, a sponge, a knife blade, a scalpel blade,a pin, a safety pin, a tip, tube, an adapter, a guide such as a cuttingguide, a measurement device and any combination thereof.

An interaction involves a relationship between at least two articles,such as a predetermined distance between the articles, a predeterminedangle between the articles, at least one article in a predeterminedorientation with respect to at least one other article, a predetermineddifference in speed between at least two articles, a predetermineddifference in velocity and any combination thereof. Two articles aretravelling at different speeds if the total distance one travels in atime interval Δt is different from the total distance the other travelsin the time interval Δt. Two articles are travelling at differentvelocities if at least one of the following is true: they are travelingat different speeds, or they are traveling in different directions.

Examples of interactions include, but are not limited to:

-   -   Holding two hands at a fixed angle with relation to each other.    -   Tracking, which typically involves keeping a constant distance        between an endoscope and at least one tool.    -   A suction tube can be kept a constant distance from an ablator,        with the longitudinal axis of the tube at a fixed angle relative        to the longitudinal axis of the ablator.    -   A grasper can be closed or kept closed if the distance between        at least a portion of the grasper and tissue is smaller than a        predetermined distance; similarly, a grasper can be opened if        the distance between at least a portion of the grasper and the        tissue is greater than a predetermined distance.    -   If two retractors are closer to tissue than a predetermined        amount and closer to each other than a different predetermined        amount, maintaining the retractors a fixed distance apart, with        the flats of their blades parallel to each other. Optionally,        the retractors can be maintained a fixed distance from the        tissue. This interaction can be used, for example, to keep an        incision in an organ open with minimum stress on the tissue,        even if the organ is moved.    -   A fluid delivery tube and a suction tube can be kept fixed        distances (which need not be the same) from a cautery, with a        fixed angle between the tip of the fluid delivery tube, the tip        of the cautery and the tip of the suction tube. In addition, the        longitudinal axes of the tubes can be at fixed angles (which        need not be the same) relative to the longitudinal axis of the        cautery    -   The speed with which a tool moves toward an organ can be kept        below a maximum.

Many more examples will be obvious to one skilled in the art.

In some embodiments, the system provides an override facility, such thatsuch that an output movement command can be overridden. The override canbe vocal, a predetermined movement, a predetermined future movement or athought indicating at least one predefined future development. Themovement or future movement can be movement of a tool, a hand, an eye,an arm, a finger, a chest, a neck, a head, a mouth, a tongue, vocalcords, a leg, a toe, a foot or any combination thereof. An actualmovement can be detected by any movement detection means, as describedherein. A future movement can be detected by means of muscular electricor magnetic patterns, or from measureable brain signals. An example ofan electrical measurement of brain signals is an electroencephalographicpattern. Similarly, an override thought can be detected by means ofbrain signals.

In some embodiments, the system can identify at least one unwantedmovement protocol for at least one moving element. Non-limiting examplesof unwanted movement protocols include: involuntary movement of a bodypart, saccadic movement of an eye, vestibulo-ocular movement of an eye,winking an eye, blinking an eye, tremor of a body part, a tic in a bodypart, myoclonus of a body part, dystonia, and any combination thereof.

In such embodiments, the preferred response is for the system to ignorethe unwanted movement, so that the actual output movement is unaffectedby and substantially independent of the unwanted movement. Fornon-limiting example, in a system where the movement of an endoscope isproportional to movement of an eye, the jerkiness of the actual eyemovement, imparted both by saccadic movement and vestibule-ocularmovement, will be “programmed out” so that the movement of the endoscopeis smooth. Similarly, if eye movement is controlling movement of anendoscope to the right, a quick glance upward will be “programmed out”;the endoscope will not diverge from the direct path to the right.

In another non-limiting example, movement of two retractors iscontrolled by movement of two arms. During a retraction to further openan incision, the operator suffers a muscular twitch that jerks an armupward. The jerk is ignored so that the retractors move apart smoothly.

In some embodiments, control of the tools and of maneuvering of thelaparoscope does not require physical contact between the operator andeither the tools or the laparoscope. In such embodiments, the system ofthe present invention can be used with the operator and the operatingteam in the same room as the patient during the operation, or the systemof the present invention can be used for remote surgery, with theoperator controlling the laparoscope and the tools from a locationremote from the patient. For example, controlling maneuvering of thelaparoscope image does not require use of a joystick or other objectwhich requires the operator or other user, during the operation, toplace his hand in contact with the device and, by those movements,control maneuvering of the laparoscope display.

In preferred embodiments, input movement commands and movement protocolsand output movement commands and movement protocols are stored in adatabase in either wired or wireless communication with a processor andthe maneuvering system.

In some embodiments of the system, the system is programmed such that itcan add new input movement commands and their associated output movementcommands and store the new input and output movement commands in adatabase. In some embodiments, at least one of new input protocols andnew output protocols can be added to at least one database.

In some variants of these embodiments, a database further comprises atleast one set of commands linkable to an operator so that, afterentering an identifier, the operator can, in some embodiments, customizeat least one input command by enterering into the database the member ofthe group consisting of an input movement protocol, an input action, aninput position and an input reposition and any combination thereofcomprising the customized input command. The customized input commandcan be associated with an output command.

In some variants of these embodiments, the operator can also customizeoutput commands by entering into a database the member of the groupconsisting of an output movement protocol, an output action, an outputposition and an output reposition and any combination thereof comprisingthe customized output command. For non-limiting example, an operator canselect a movement protocol of a spiral movement away from himself as theinput movement command to start a flow of fluid to clean an endoscope'soptics. In this example, the operator can select a spiral movementtowards himself for the command to stop the flow of fluid.

It should be noted that maneuvering of the laparoscope display can beaccomplished by physical maneuvering (physically moving some portion ofthe laparoscope or the imaging optics), by virtual maneuvering (changingthe viewed image by means of computer algorithms that alter the portionof the field of view which is displayed), or by some combinationthereof.

Another possible use for the system of the present invention is forstudy or training. By use of a plurality of display screens, a number ofstudents can observe the operation in real time; the students can be inlocations remote from both the patient and the operator. The displayview can be marked to assist the students in understanding what ispresent in the display view. Marking can be done by persons other thanthe operator; if desired, in some embodiments, the operator need not beaware of marks applied to the display view for study or teachingpurposes.

In some embodiments, a system and method is provided for providingaugmented reality images of a field of view, where the augmentation canbe images provided by another imaging modality, stored images or otherstored data, information entered by a user, and any combination thereof.

The field of view can be the field of view of an endoscope orlaparoscope, or the field of view of a surgical tool.

In some embodiments, control of maneuvering is via a body-mounted userinterface which comprises at least one sensor, the sensor configured tosense at least one parameter associated with body motion, with the bodymotion directing maneuvering of a displayed image, of the endoscope, ofa tool, and any combination thereof.

The sensor or sensors can be placed in conjunction with a body portionof an operator, or can be placed so that a body portion of the operatoris within range of the sensor.

A sensor can be, for non-limiting example, an ultrasound sensor, an IRsensor, a heat sensor, a pressure sensor, a current sensor, anaccelerometer, a tilt sensor, a movement sensor, a gyroscope, aninertial sensor, a goniometer, a magnetometer, a strain sensor, anelectroencephalographic sensor, an electrical sensor, a magnetic sensor,a position sensor configured to determine position of said at least oneportion of said human body; a speed sensor configured to determinevelocity of said at least one portion of said human body; anaccelerometer configured to determine acceleration of said at least oneportion of said human body; a camera and image detection softwareconfigured to determine at least one of position, velocity andacceleration of said at least one portion of said human body; an RFsensor and RF emitter coupleable to said at least one portion of saidhuman body configured to determine at least one of position, velocityand acceleration of said at least one portion of said human body; asound sensor and sound emitter coupleable to said at least one portionof said human body configured to determine at least one of position,velocity and acceleration of said at least one portion of said humanbody; an electroencephalographic sensor configured to determine, from atleast one electroencephalographic pattern, at least one parameterassociated with at least one of position, velocity and acceleration ofsaid at least one portion of said human body, a sensor configured todetermine, from at least one electrical pattern in at least one muscle,at least one parameter associated with at least one of position,velocity and acceleration of said at least one portion of said humanbody and any combination thereof.

The sensors are preferably MEMS devices.

The sensors are preferably in wireless communication with the dataprocessor controlling maneuvering of the display view.

As a non-limiting example of an embodiment, accelerometers can becomprised in a band encircling the operator's lower arm, thereby sensingmovement of arm.

In some embodiments, the sensor or sensors can comprise viewing means,such as, but not limited to a camera, an IR sensor, an ultrasoundsensor, a sound sensor, an RF sensor, a heat sensor, an electricalsensor, a magnetic sensor, and any combination thereof. In such sensors,the viewing means senses either movement of the body portion or patternsassociated with movement of the body portion. The detectable movementcan include speech, which can be detected by a sound sensor.

Other possible sensors include, but are not limited to: a positionsensor configured to determine position of said at least one portion ofsaid human body; a speed sensor configured to determine velocity of saidat least one portion of said human body; an accelerometer configured todetermine acceleration of said at least one portion of said human body;a camera and image detection software configured to determine at leastone of position, velocity and acceleration of said at least one portionof said human body; an RF sensor and RF emitter coupleable to said atleast one portion of said human body configured to determine at leastone of position, velocity and acceleration of said at least one portionof said human body; a sound sensor and sound emitter coupleable to saidat least one portion of said human body configured to determine at leastone of position, velocity and acceleration of said at least one portionof said human body; an electroencephalographic sensor configured todetermine, from at least one electroencephalographic pattern, at leastone parameter associated with at least one of position, velocity andacceleration of said at least one portion of said human body, a sensorconfigured to determine, from at least one electrical pattern in atleast one muscle, at least one parameter associated with at least one ofposition, velocity and acceleration of said at least one portion of saidhuman body and any combination thereof.

Non-limiting examples of body portions whose activity can be sensed by asensor include at least a portion of any of the following: a finger, ahand, a wrist, a forearm, an elbow, a shoulder, an arm, a toe, a foot, aleg, a neck, a chest, an abdomen, a torso or trunk, a head, an eye, themouth, the brain, and a face. The sensors can comprise a portion of aglove, a band, a harness or a mask or be mounted on or in a glove, aband, a harness or a mask.

A glove can be fingerless, or can have one or more fingers. It can behand-length, wrist-length, elbow length, can extend partway up the arm,or can extend all the way up the arm. It can have any combination oflength and number of fingers. One or two gloves can be worn; they cancomprise any combination of the above features.

Non-limiting examples of bands include elastic bands and non-elasticbands; bands are preferably flexible in order to conform to the surfaceof the body portion, but portions of the band can be rigid. The band canbe continuous or can comprise at least one break. Bands can compriseties, buckles, or any other closure means or size-adjustment means knownin the art. They can be fixed-length or variable-length. The band can beof any desired width, up to one that covers the entire arm and even partof the hand. There can therefore be overlap between what is considered a“glove” and what is considered an “arm-covering band”.

Bands can comprise armbands, hand bands, face bands and chest bands.Chest-movement sensors can be comprised in a harness, which can beelastic or non-elastic and which can stretch to fit over the headwithout need for additional closures, can comprise one or more closures,can comprise one or more length-adjustment mechanisms, and anycombination thereof. Closures and length-adjustment mechanisms can beties, buckles, any other closure mechanism known in the art and anycombination thereof.

In some embodiments, the intended movement can be detectedencephalographically, via at least one sensor, preferably on the head,configured to determine, from at least one electroencephalographicpattern, at least one parameter associated with at least one ofposition, velocity and acceleration of at least one portion of a humanbody. The intended movement can include speech; in this case, theelectroencephalographic pattern can be a pattern indicating activity ofthe brain speech centers.

In some embodiments, the detectable electroencephalographic pattern caninclude a pattern indicative of alarm or fright. Such a pattern can beused, for non-limiting example, as an override signal.

In some embodiments, at least one electric or magnetic sensor detectselectrical and/or magnetic patterns associated with movement of at leastone muscle. From these electrical and/or magnetic patterns, the intendedmovement of the muscle and, therefore, the intended movement of the bodyportion can be determined and translated into an intended input movementprotocol of the surgical tool. The sensor can be remote from the bodyportion intended to be moved; for example, electrical patterns measuredfor one or more chest muscles can be used to determine intended movementof an arm and, thence, the desired movement of a surgical tool.

Any combination of the above sensors can be used to maneuver a surgicaltool. The tool maneuver can be a maneuver generated by the system inresponse to a detected movement protocol, a maneuver directly commandedby a user and any combination thereof.

There can be one viewing means per tool, one viewing means can view aplurality of tools, a plurality of viewing means can view one tool, oneviewing means can view a plurality of tools, a plurality of viewingmeans can view a plurality of tools, and any combination thereof.

For non-limiting example, to move the center of the display view towardsthe right of the display, the operator gestures rightward. An upwardgestures zooms the display view outward, shrinking objects in view; agesture away from the body moves the center of the display view towardsthe top of the display, and any combination thereof. Other gestures cancontrol returning to a previous view or selecting an object, such as atool, to be tracked, where following an object means keeping theselected object at the center of the field of view and, if possible,keeping constant its apparent size in the display view.

In another non-limiting example of an embodied gesture, shaking a tooltip selects the tool as the object to be tracked. This informs thesystem that the shaken tool is to be tracked; it is to be kept in thecenter of the field of view. A second shake of the tool tip stopstracking. Shaking another tool transfers tracking to the shaken tool.

Another non-limiting example of an embodied gesture is opening andclosing the hand to open and close a grasper or bringing the thumbtowards a finger to close a grasper and separating the thumb and afinger to open a grasper.

The gesture embodiments described hereinabove can be used in anycombination.

Gestures can be combined with, for example, use of a touchscreen orprepared surface. In such embodiments, the operator can touch the imageon a screen or other prepared surface to select an object, then executea gesture to indicate what the object is to do. For non-limitingexample, in such an embodiment, in order to retract tissue seen near thetop of the screen with a grasper seen near the right side of the screen,the operator touches the image of the grasper to select the grasper andgestures leftward and away from himself. When the tip of the grasper isabove the tissue to be retracted, the operator gestures downward andopens his hand, thereby opening the grasper and moving its tip downtowards the tissue. When one grasper jaw is above the tissue and onebelow, a gesture away from himself moves the grasper jaws around thetissue and closure of the hand closes the grasper, grasping the tissue.Another gesture then retracts the tissue. The operator can then touchthe image of the grasper again, to stop tracking of the grasper, whichfixes the grasper in position.

In a variant of this embodiment, when the grasper is in position,instead of closing his hand, the operator touches a predeterminedposition on the touchscreen and the grasper closes. The operator canthen move a hand, as described above, to reposition the grasper.

Other embodiments of gestures of tool movement and of means ofidentifying tools will be obvious to one skilled in the art.

In some embodiments, for at least one input movement protocol, theoutput movement protocol is such that the movement of the tool isproportional to the movement of the body portion, with larger movementsof the body portion resulting in proportionally larger movements of thetool. The magnitude of the constant of proportionality can differ fordifferent input movement protocols, The constant of proportionality canbe much less than 1, so that relatively large movements of the bodyportion result in small movements of the tool. The constant ofproportionality can be 1, so that the magnitude of the output movementis substantially identical to the magnitude of the movement of the bodyportion. The constant of proportionality can be greater than 1, so thatthe magnitude of the output movement is greater than the magnitude ofthe movement of the body portion.

It is also possible for the magnitude of an output movement to beindependent of the magnitude of the movement of the body portion. Fornon-limiting example, if a movement of a body portion is associated witha command to activate a tool, then the size of the movement isirrelevant; a tool is either active or it is not.

In some embodiments, for at least one input movement protocol, theoutput movement protocol is such that the movement of the tool issubstantially identical to the movement of the body portion.

In some embodiments, for at least one input movement protocol, theoutput movement protocol is a fixed movement of a tool. For example, anopening movement of the hand, whether large or small, causes a grasperto open fully.

Any combination of proportional movement, identical movement, and fixedmovement can be used for the output protocols.

The movement of an endoscope or other surgical tool can be parallel tothe X axis; parallel to the Y axis; parallel to the Z-axis; rotationaround an axis parallel to the X axis; rotation around an axis parallelto the Y axis; rotation around an axis parallel to the Z axis; and anycombination thereof.

In embodiments of the system wherein movement of a surgical tool iscontrolled by movement of a body portion, whether sensed as movement ofthe body portion or sensed as movement of a surgical tool, movement ofthe surgical tool need not be in the same direction as the movement ofthe body portion. For example, a movement left can translate intomovement upward of the surgical tool, rather than moving the bodyportion upward to move the surgical tool upward. The direction ofmovement of the surgical tool can be any of: movement of the bodyportion in a direction parallel to the X axis translates to movement ofthe surgical tool in a direction parallel to the X axis, movement of thebody portion in a direction parallel to the X axis translates tomovement of the surgical tool in a direction parallel to the Y axis,movement of the body portion in a direction parallel to the X axistranslates to movement of the surgical tool in a direction parallel tothe Z axis, movement of the body portion in a direction parallel to theY axis translates to movement of the surgical tool in a directionparallel to the X axis, movement of the body portion in a directionparallel to the Y axis translates to movement of the surgical tool in adirection parallel to the Y axis, movement of the body portion in adirection parallel to the Y axis translates to movement of the surgicaltool in a direction parallel to the Z axis, movement of the body portionin a direction parallel to the Z axis translates to movement of thesurgical tool in a direction parallel to the X axis, movement of thebody portion in a direction parallel to the Z axis translates tomovement of the surgical tool in a direction parallel to the Y axis,movement of the body portion in a direction parallel to the Z axistranslates to movement of the surgical tool in a direction parallel tothe Z axis, rotation of the body portion about an axis parallel to the Xaxis translates to rotation of the surgical tool about an axis parallelto the X axis, rotation of the body portion about an axis parallel tothe X axis translates to rotation of the surgical tool about an axisparallel to the Y axis, rotation of the body portion about an axisparallel to the X axis translates to rotation of the surgical tool aboutan axis parallel to the Z axis, rotation of the body portion about anaxis parallel to the Y axis translates to rotation of the surgical toolabout an axis parallel to the X axis, rotation of the body portion aboutan axis parallel to the Y axis translates to rotation of the surgicaltool about an axis parallel to the Y axis, rotation of the body portionabout an axis parallel to the Y axis translates to rotation of thesurgical tool about an axis parallel to the Z axis, rotation of the bodyportion about an axis parallel to the Z axis translates to rotation ofthe surgical tool about an axis parallel to the X axis, rotation of thebody portion about an axis parallel to the Z axis translates to rotationof the surgical tool about an axis parallel to the Y axis, rotation ofthe body portion about an axis parallel to the Z axis translates torotation of the surgical tool about an axis parallel to the Z axis, andany combination thereof.

In some embodiments, linear movement of the body portion, whether sensedas movement of the body portion or sensed as movement of a surgicaltool, is translated to rotational movements of the endoscope or othersurgical tool. For example: movement of the body portion in a directionparallel to the X axis translates to rotation of the surgical tool aboutan axis parallel to the X axis, movement of the body portion in adirection parallel to the X axis translates to rotation of the surgicaltool about an axis parallel to the Y axis, movement of the body portionin a direction parallel to the X axis translates to rotation of thesurgical tool about an axis parallel to the Z axis, movement of the bodyportion in a direction parallel to the Y axis translates to rotation ofthe surgical tool about an axis parallel to the X axis, movement of thebody portion in a direction parallel to the Y axis translates torotation of the surgical tool about an axis parallel to the Y axis,movement of the body portion in a direction parallel to the Y axistranslates to rotation of the surgical tool about an axis parallel tothe Z axis, movement of the body portion in a direction parallel to theZ axis translates to rotation of the surgical tool about an axisparallel to the X axis, movement of the body portion in a directionparallel to the Z axis translates to rotation of the surgical tool aboutan axis parallel to the Y axis, movement of the body portion in adirection parallel to the Z axis translates to rotation of the surgicaltool about an axis parallel to the Z axis and any combination thereof.

In some embodiments, rotational movement of the body portion, whethersensed as movement of the body portion or sensed as movement of asurgical tool, is translated to linear movements of the surgical tool.For example: rotation of the body portion about an axis parallel to theX axis translates to movement of the surgical tool in a directionparallel to the X axis, rotation of the body portion about an axisparallel to the X axis translates to movement of the surgical tool in adirection parallel to the Y axis, rotation of the body portion about anaxis parallel to the X axis translates to movement of the surgical toolin a direction parallel to the Z axis, rotation of the body portionabout an axis parallel to the Z axis translates to movement of thesurgical tool in a direction parallel to the X axis, rotation of thebody portion about an axis parallel to the Z axis translates to movementof the surgical tool in a direction parallel to the Y axis, rotation ofthe body portion about an axis parallel to the Z axis translates tomovement of the surgical tool in a direction parallel to the Z axis andany combination thereof.

Any combination of the above translations can be used in an embodiment.

In some embodiments, a predetermined output protocol is configured todetermine allowed and restricted movements of the endoscope fromhistorical movements of the endoscope according with historical movementpatterns of at least one surgical tool in at least one previous surgery.Thus, according to these embodiments, the predetermined protocolcomprises a communicable database storing each 3D spatial position ofthe endoscope according with at least two 3D spatial positions, thecurrent 3D spatial position, 3D_(current), and at least one previous 3Dspatial position, 3D_(previous), of at least one surgical tool, suchthat each movement pattern of the at least one surgical tool and each 3Dposition of the endoscope according with the same is stored; thepredetermined protocol is configured to determine allowed and restrictedmovements of the endoscope from the stored movement patterns of the atleast one surgical tool and the stored movements of the endoscope, suchthat the allowed movements of the endoscope are movements in which theendoscope is located substantially in at least one of the endoscope 3Dspatial positions according with at least one 3D tool movement pattern,and the restricted movements are movements in which the location of theendoscope is substantially different from the n 3D endoscope spatialpositions according with the n movement patterns.

The system further comprises a predetermined set of rules to controlmovement of the surgical tool. As described hereinbelow, the rules,among other functions, ensure that a surgical tool can be moved withoutundesired contact with another surgical tool or with a portion of thebody. The predetermined set of rules is configured to take intoconsideration all the possible factors which may be important during thesurgical procedure. The predetermined set of rules can comprise anycombination of the following rules:

-   -   a. a route rule;    -   b. an environment rule;    -   c. an operator input rule;    -   d. a proximity rule;    -   e. a collision prevention rule;    -   f a history based rule;    -   g. a tool-dependent allowed and restricted movements rule.    -   h. a most used tool rule;    -   i. a right tool rule;    -   j. a left tool rule;    -   k. a field of view rule;    -   l. a no fly zone rule;    -   m. an operator input rule;    -   n. a preferred volume zone rule;    -   o. a preferred tool rule;    -   p. a movement detection rule,

Thus, for example, the collision prevention rule defines a minimumdistance below which two or more tools should not be brought together(i.e., there is minimum distance between two or more tools that shouldbe maintained). If the movement of one tool will cause it to comedangerously close to another tool (i.e., the distance between them,after the movement, is smaller than the minimum distance defined by thecollision prevention rule), the controller either alerts the user thatthe movement is a restricted movement or does not permit the movement.

It should be emphasized that all of the above (and the followingdisclosure) is enabled by constantly monitoring the surgicalenvironment, and identifying and locating the 3D spatial location ofeach element/tool in the surgical environment.

The identification is provided by conventional means known to anyskilled in the art (e.g., image processing, optical means etc.).

The following provides explanations for each of the above mentionedrules and their functions:

According to some embodiments, the route rule comprises a predefinedroute in which the at least one surgical tool is configured to movewithin the surgical environment; the allowed movements are movements inwhich the at least one surgical tool is located within the borders ofthe predefined route, and the restricted movements are movements inwhich the at least one surgical tool is located out of the borders ofthe predefined route. Thus, according to this embodiment, the route rulecomprises a communicable database storing at least one predefined routein which the at least one surgical tool is configured to move within thesurgical environment; the predefined route comprises n 3D spatialpositions of the at least one surgical tool in the route; n is aninteger greater than or equal to 2; allowed movements are movements inwhich the at least one surgical tool is located substantially in atleast one of the n 3D spatial positions of the predefined route, andrestricted movements are movements in which the location of the at leastone surgical tool is substantially different from the n 3D spatialpositions of the predefined route.

In other words, according to the route rule, each of the surgical tool'scourses (and path in any surgical procedure) is stored in a communicabledatabase. Allowed movements are defined as movements in which the atleast one surgical tool is located substantially in at least one of thestored routes; and restricted movements are movements in which the atleast one surgical tool is in a substantially different location thanany location in any stored route.

According to some embodiments, the environmental rule is configured todetermine allowed and restricted movements according to hazards orobstacles in the surgical environment as received from an endoscope orother sensing means. Thus, according to this embodiment, theenvironmental rule comprises a communicable database; the communicabledatabase is configured to received real-time images of the surgicalenvironment and is configured to perform real-time image processing ofthe same and to determine the 3D spatial position of hazards orobstacles in the surgical environment; the environmental rule isconfigured to determine allowed and restricted movements according tohazards or obstacles in the surgical environment, such that restrictedmovements are movements in which at least one surgical tool is locatedsubstantially in at least one of the 3D spatial positions, and allowedmovements are movements in which the location of at least one surgicaltool is substantially different from the 3D spatial positions.

In other words, according to the environment rule, each element in thesurgical environment is identified so as to establish which is a hazardor obstacle (and a path in any surgical procedure) and each hazard andobstacle (and path) is stored in a communicable database. Restrictedmovements are defined as movements in which the at least one surgicaltool is located substantially in the same location as that of thehazards or obstacles; and the allowed movements are movements in whichthe location of the at least one surgical tool is substantiallydifferent from that of all of the hazards or obstacles.

According to other embodiments, hazards and obstacles in the surgicalenvironment are selected from a group consisting of tissues, surgicaltools, organs, endoscopes and any combination thereof.

According to some embodiments, the operator input rule is configured toreceive an input from the operator of the system regarding the allowedand restricted movements of the at least one surgical tool. Thus,according to this embodiment, the operator input rule comprises acommunicable database; the communicable database is configured toreceive an input from the operator of the system regarding allowed andrestricted movements of the at least one surgical tool.

According to other embodiments, the input comprises n 3D spatialpositions; n is an integer greater than or equal to 2; wherein at leastone of which is defined as an allowed location and at least one of whichis defined as a restricted location, such that the allowed movements aremovements in which the at least one surgical tool is locatedsubstantially in at least one of the n 3D allowed spatial positions, andthe restricted movements are movements in which the location of the atleast one surgical tool is substantially different from the n 3D allowedspatial positions.

According to other embodiments, the input comprises at least one ruleaccording to which allowed and restricted movements of the at least onesurgical tool are determined, such that the spatial position of the atleast one surgical tool is controlled by the controller according to theallowed and restricted movements.

According to other embodiments, the operator input rule can convert anallowed movement to a restricted movement and a restricted movement toan allowed movement.

According to some embodiments, the proximity rule is configured todefine a predetermined distance between the at least one surgical tooland at least one another surgical tool; the allowed movements aremovements which are within the range or out of the range of thepredetermined distance, and the restricted movements which are out ofthe range or within the range of the predetermined distance; the allowedmovements and the restricted movements are defined according todifferent ranges. Thus, according to this embodiment, the proximity ruleis configured to define a predetermined distance between at least twosurgical tools. In a preferred embodiment, the allowed movements aremovements which are within the range of the predetermined distance,while the restricted movements which are out of the range of thepredetermined distance. In another preferred embodiment, the allowedmovements are movements which are out of the range of the predetermineddistance, while the restricted movements are within the range of thepredetermined distance

It should be pointed out that the above mentioned distance can beselected from the following:

-   -   (a) the distance between the tip of the first tool and the tip        of the second tool;    -   (b) the distance between the body of the first tool and the tip        of the second tool;    -   (c) the distance between the body of the first tool and the body        of the second tool;    -   (d) the distance between the tip of the first tool and the body        of the second tool; and any combination thereof.

According to another embodiment, the proximity rule is configured todefine a predetermined angle between at least three surgical tools;allowed movements are movements which are within the range or out of therange of the predetermined angle, and restricted movements are movementswhich are out of the range or within the range of the predeterminedangle.

According to some embodiments, the collision prevention rule isconfigured to define a predetermined distance between the at least onesurgical tool and an anatomical element within the surgical environment(e.g. tissue, organ, another surgical tool or any combination thereof);the allowed movements are movements which are in a range that is largerthan the predetermined distance, and the restricted movements aremovements which is in a range that is smaller than the predetermineddistance.

According to another embodiment, the anatomical element is selected froma group consisting of tissue, organ, another surgical tool or anycombination thereof.

According to some embodiments, the surgical tool is an endoscope. Theendoscope is configured to provide real-time images of the surgicalenvironment.

According to some embodiments, the right tool rule is configured todetermine the allowed movement of the endoscope according to themovement of a surgical tool in a specified position in relation to theendoscope, preferably positioned to right of the same. According to thisrule, the tool which is defined as the right tool is constantly trackedby the endoscope. According to some embodiments, the right tool isdefined as the tool positioned to the right of the endoscope; accordingto other embodiments, any tool can be defined as the right tool. Anallowed movement, according to the right tool rule, is a movement inwhich the endoscope field of view is moved to a location substantiallythe same as the location of the right tool, thereby tracking the righttool. A restricted movement, according to the right tool rule, is amovement in which the endoscope field of view is moved to a locationsubstantially different from the location of the right tool.

According to some embodiments, the left tool rule is configured todetermine the allowed movement of the endoscope according to themovement of a surgical tool in a specified position in relation to theendoscope, preferably positioned to left of the same. According to thisrule, the tool which is defined as the left tool is constantly trackedby the endoscope. According to some embodiments, the left tool isdefined as the tool positioned to the left of the endoscope; accordingto other embodiments, any tool can be defined as the left tool. Anallowed movement, according to the left tool rule, is a movement inwhich the endoscope field of view is moved to a location substantiallythe same as the location of the left tool. A restricted movement,according to the left tool rule, is a movement in which the endoscopefield of view is moved to a location substantially different from thelocation of the left tool.

According to some embodiments, the field of view rule is configured todefine a field of view and maintain that field of view. The field ofview rule is defined such that if the endoscope is configured to track apredetermined set of tools in a desired field of view, when one of thosetools is no longer in the field of view, the rule instructs theendoscope to zoom out so as to reintroduce the tool into the field ofview. Thus, according to this embodiment, the field of view rulecomprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equal to 2; the combination of all of then 3D spatial positions provides a predetermined field of view; the fieldof view rule is configured to determine the allowed movement of theendoscope within the n 3D spatial positions so as to maintain a constantfield of view, such that the allowed movements are movements in whichthe endoscope is located substantially in at least one of the n 3Dspatial positions, and the restricted movements are movements in whichthe location of the endoscope is substantially different from the n 3Dspatial positions.

Thus, according to another embodiment of the field of view rule, thefield of view rule comprises a communicable database comprising n 3Dspatial positions; n is an integer greater than or equal to 2; thecombination of all of the n 3D spatial positions provides apredetermined field of view. The field of view rule further comprises acommunicable database of m tools and the 3D spatial locations of thesame, where m is an integer greater than or equal to 1 and where a toolcan be a surgical tool, an anatomical element and any combinationthereof. The combination of all of the n 3D spatial positions provides apredetermined field of view. The field of view rule is configured todetermine allowed movement of the endoscope such that the m 3D spatialpositions of the tools comprise at least one of the n 3D spatialpositions of the field of view, and restricted movements are movementsin which the 3D spatial position of at least one tool is substantiallydifferent from then 3D spatial positions of the field of view.

According to another embodiment, the preferred volume zone rulecomprises a communicable database comprising n 3D spatial positions; nis an integer greater than or equal to 2; the n 3D spatial positionsprovides the preferred volume zone; the preferred volume zone rule isconfigured to determine the allowed movement of the endoscope within then 3D spatial positions and restricted movement of the endoscope outsidethe n 3D spatial positions, such that the allowed movements aremovements in which the endoscope is located substantially in at leastone of the n 3D spatial positions, and the restricted movements aremovements in which the location of the endoscope is substantiallydifferent from the n 3D spatial positions. In other words, the preferredvolume zone rule defines a volume of interest (a desired volume ofinterest), such that an allowed movement, according to the preferredvolume zone rule, is a movement in which the endoscope (or any surgicaltool) is moved to a location within the defined preferred volume. Arestricted movement, according to the preferred volume zone rule, is amovement in which the endoscope (or any surgical tool) is moved to alocation outside the defined preferred volume.

According to another embodiment, the preferred tool rule comprises acommunicable database, the database stores a preferred tool; thepreferred tool rule is configured to determine the allowed movement ofthe endoscope according to the movement of the preferred tool. In otherwords, the preferred tool rule defines a preferred tool (i.e., a tool ofinterest) that the user of the system wishes to track. An allowedmovement, according to the preferred tool rule, is a movement in whichthe endoscope is moved to a location substantially the same as thelocation of the preferred tool. A restricted movement is a movement inwhich the endoscope is moved to a location substantially different fromthe location of the preferred tool. Thus, according to the preferredtool rule the endoscope constantly tracks the preferred tool, such thatthe field of view, as seen from the endoscope, is constantly thepreferred tool. It should be noted that the user may define in thepreferred tool rule to constantly track the tip of a preferred tool oralternatively, the user may define the preferred tool rule to constantlytrack the body or any location on the preferred tool.

According to some embodiments, the no fly zone rule is configured todefine a restricted zone into which no tool (or alternatively nopredefined tool) is permitted to enter. Thus, according to thisembodiment, the no fly zone rule comprises a communicable databasecomprising n 3D spatial positions; n is an integer greater than or equalto 2; the n 3D spatial positions define a predetermined volume withinthe surgical environment; the no fly zone rule is configured todetermine a restricted movement if the movement is within the no flyzone and an allowed movement if the movement is outside the no fly zone,such that restricted movements are movements in which the at least onesurgical tool is located substantially in at least one of the n 3Dspatial positions, and the allowed movements are movements in which thelocation of the at least one surgical tool is substantially differentfrom the n 3D spatial positions.

According to another embodiment, the most used tool function isconfigured to define (either real-time, during the procedure or prior tothe procedure) which tool is the most used tool (i.e., the tool which ismoved the most during the procedure) and to instruct the maneuveringsubsystem to constantly position the endoscope to track the movement ofthis tool. Thus, according to this embodiment, the most used tool rulecomprises a communicable database counting the number of movements ofeach of the surgical tools; the most used tool rule is configured toconstantly position the endoscope to track the movement of the surgicaltool with the largest number of movements. In another embodiment of themost used tool function, the communicable database measures the amountof movement of each of the surgical tools; the most used tool rule isconfigured to constantly position the endoscope to track the movement ofthe surgical tool with the largest amount of movement.

According to another embodiment, the system is configured to alert thephysician of a restricted movement of at least one surgical tool. Thealert can be audio signaling, voice signaling, light signaling, flashingsignaling and any combination thereof.

According to another embodiment, an allowed movement is one permitted bythe controller and a restricted movement is one denied by thecontroller.

According to another embodiment, the operator input rule function isconfigured to receive an input from the operator of the system regardingallowed and restricted movements of the at least one surgical tool. Inother words, the operator input rule function receives instructions fromthe physician as to what can be regarded as allowed movements and whatare restricted movements. According to another embodiment, the operatorinput rule is configured to convert an allowed movement to a restrictedmovement and a restricted movement to an allowed movement.

According to some embodiments, the history-based rule is configured todetermine the allowed and restricted movements according to historicalmovements of the at least one surgical tool in at least one previoussurgery. Thus, according to this embodiment, the history-based rulecomprises a communicable database storing each 3D spatial position ofeach of the surgical tools, such that each movement of each surgicaltool is stored; the history-based rule is configured to determineallowed and restricted movements according to historical movements ofthe at least one surgical tool, such that the allowed movements aremovements in which the at least one surgical tool is locatedsubstantially in at least one of the 3D spatial positions, and therestricted movements are movements in which the location of the at leastone surgical tool is substantially different from the n 3D spatialpositions.

According to some embodiments, the tool-dependent allowed and restrictedmovements rule is configured to determine allowed and restrictedmovements according to predetermined characteristics of the surgicaltool, where the predetermined characteristics of the surgical tool areselected from a group consisting of: physical dimensions, structure,weight, sharpness, and any combination thereof. Thus, according to thisembodiment, the tool-dependent allowed and restricted movements rulecomprises a communicable database; the communicable database isconfigured to store predetermined characteristics of at least one of thesurgical tools; the tool-dependent allowed and restricted movements ruleis configured to determine allowed and restricted movements according tothe predetermined characteristics of the surgical tool.

According to another embodiment, the predetermined characteristics ofthe surgical tool are selected from a group consisting of: physicaldimensions, structure, weight, sharpness, and any combination thereof.

According to this embodiment, the user can define, e.g., the structureof the surgical tool he wishes the endoscope to track. Thus, accordingto the tool-dependent allowed and restricted movements rule theendoscope constantly tracks the surgical tool having the predeterminedcharacteristics as defined by the user.

According to another embodiment of the present invention, the movementdetection rule comprises a communicable database comprising thereal-time 3D spatial positions of each surgical tool; the movementdetection rule is configured to detect movement of at least one surgicaltool. When a change in the 3D spatial position of that surgical tool isreceived, allowed movements are movements in which the endoscope isre-directed to focus on the moving surgical tool.

According to some embodiments, the at least one location estimatingmeans is at least one endoscope configured to acquire real-time imagesof a surgical environment within the human body for the estimation ofthe location of at least one surgical tool.

According to another embodiment, the location estimating means compriseat least one selected from a group consisting of optical imaging means,radio frequency transmitting and receiving means, at least one mark onat least one surgical tool and any combination thereof.

According to another embodiment, the at least one location estimatingmeans is an interface subsystem between a operator and at least onesurgical tool, the interface subsystem comprising (a) at least one arraycomprising N regular light sources or N pattern light sources, where Nis a positive integer; (b) at least one array comprising M cameras,where M is a positive integer; (c) optional optical markers and meansfor attaching the optical markers to at least one surgical tool; and (d)a computerized algorithm operable via the controller, the computerizedalgorithm configured to record images received by each camera of each ofthe M cameras and to calculate therefrom the position of each of thetools, and further configured to provide automatically the results ofthe calculation to the human operator of the interface.

In some embodiments of the system, the faster the body portion is moved,the faster the selected portion of the surgical tool moves. In theseembodiments, the system provides a warning if the speed is above apredetermined maximum. Examples of the method of warning include, butare not limited to, a constant volume tone, a constant pitch tone, avarying volume tone, a varying pitch tone, a vocal signal, a constantcolor visual signal, a constant brightness visual signal, a varyingcolor visual signal, a varying brightness visual signal, a signalvisible on at least some part of the endoscope image, a signal visibleon at least some portion of the patient, a signal visible in at leastsome portion of the surroundings of the patient, a vibration in thecontrol unit, a temperature change in the control unit, and anycombination of the above.

According to some embodiments of the present invention, the velocity ofthe surgical tool's movement will be adjusted as a function of thedistance of the tool tip from the organ\tissue. For non-limitingexample, the closer the tip of an endoscope is to an organ, the slowerthe endoscope moves, thereby, on the one hand, helping ensure that theendoscope tip stops in a desired position and, on the other hand,reducing the probability that the endoscope will contact theorgan/tissue, either through overshoot or through a miscalculation suchas could occur from drift in the system.

In some embodiments, the display comprises augmented reality elements.

In some embodiments, the operator can mark a point or points in thebody. These points can indicate an organ or tissue, be a location on anorgan or tissue, be a location within the body not on an organ ortissue, indicate a tool or other object (such as a swab) introduced bythe operator, or be a location (such as a tool tip) on a tool or otherobject.

Sets of points, such as but not limited to a set of points forming theoutline of an object or the surface of an object can also be marked. Anon-limiting example of an outline would be a line indicating theapproximate extent of a tumor.

Marking can be by means of touching the point on a touchscreen or otherprepared surface, touching the position of the point in a 3D display,touching a symbol representing the object on a touchscreen or preparedsurface, directing an indicator to the point by means of gestures orpredetermined sounds, any other means known in the art of specifying adesired point, and any combination thereof.

After marking, the point can be labeled; the point is indicated in theimage by a virtual marker. The virtual marker can comprise any means oflabeling images known in the art. Non-limiting examples of virtualmarkers include a predetermined geometrical shape, a predetermined word,a line encircling the image of a selected object, highlighting of theselected object (placing a patch of predetermined color or predeterminedtexture), and any combination thereof. Color-coding, with differentcolors indicating different types of virtual marker, can be used, eitheralone or in combination with any of the virtual markers described above.

In some embodiments, the virtual marker indicates a selectable displayview. In such embodiments, selection of the marker automatically altersthe display view to the view specified by the marker. Such selectabledisplay view markers can comprise, for non-limiting example, an outlineof the selectable view, a point at the center of the selectable view, apatch or different color or texture covering the selectable view, andany combination thereof.

In some embodiments, portions of the image are enhanced, typically inorder to be seen or identified more easily. Objects which can beenhanced include, but are not limited to, blood vessels, organs,ligaments, limbs and any combination thereof.

Enhancement can include, but is not limited to, increasing brightness,altering color, applying color or texture patches, and any combinationthereof.

Markers can comprise a distance or angle measurement. For non-limitingexample, the user can select two points within the display field andinstruct the system to measure the distance between the points. A markerthen indicates the two points and the distance between them. Similarly,for non-limiting example, selection of three points instructs the systemto measure the angle formed by the three points and to provide a markershowing the points and the angle they form. Any distance or anglemeasurement known in the art, such as, but not limited to, thosetypically found in Computer Aided Design (CAD) systems can beimplemented the system of the present invention. Distances and anglesmeasurements are 3D measurements. The distance marker will typically belabeled with the total distance between the start and end points. Insome embodiments, the distance marker can give the distance between theend points as a triple of values, typically the three distances (x, y,z) of a Euclidean coordinate system. Other typical coordinate systemsinclude, but are not limited to, cylindrical coordinate systems (r, θ,z) and spherical coordinate systems (r, θ, ϕ).

In some embodiments, orientation marking is provided. The orientationmarker indicates a direction fixed relative to the region of interest.Therefore, the operator can remain aware of the orientation of thedisplay view relative to the region of interest in the body, whateverthe relative orientations of the body and the display view.

In preferred embodiments, the orientation marker remains within a fixedregion in the display view. A non-limiting example of an orientationmarker is axes of a 3D coordinate system, with the axes labeled so thatthe identity of each axis is discernable at a glance. The axes are in acorner of the display view and rotate as the orientation of the displayview changes.

Another embodiment of an orientation marker comprises an arrow with afixed center, the direction of the arrow indicating a fixed (3D)direction in space. The point of the arrow will rotate around the centeras the display view changes, while the color or texture of the arrowindicates whether the fixed direction is above or below the plane of thedisplay image and the length of the arrow indicates the angle betweenthe fixed direction and the plane of the display view.

Any orientation marker known in the art can be used.

In some embodiments, the display image combines the laparoscope imagewith an image from at least one other imaging modality. The otherimaging modality can be any imaging modality known in the art, fornon-limiting example, CT, MRI, PET, ultrasound, IR imaging, heatimaging, a still camera, a videocamera, image-generation software,image-manipulation software, display of stored images, and anycombination thereof. In preferred embodiments, all images are registeredso that like portions correspond with each other and so appear to beviewed from the same distance and angle. For non-limiting example, theboundaries of the liver from an MRI scan would overlap the boundaries ofthe liver from the laparoscope image.

The images from the second imaging modality can be 2D images, 3D imagesand any combination thereof.

An image from another imaging modality can be a real-time image or canbe a stored image. For non-limiting example, the interior of the abdomencan be simultaneously imaged by ultrasound and by the laparoscope duringa procedure, with the images from the two modalities registered anddisplayed simultaneously.

In another non-limiting example, 3D MRI images of the abdomen can bemade prior to the procedure. During the procedure, the stored MRI imagesare registered with 3D structured light images from the laparoscope,providing the operator with an enhanced 3D view, in which the visibilityof blood vessels and of tumors has been increased.

In some embodiments, the laparoscope optics comprise at least onewide-angle lens, so that the field of view of the camera comprisessubstantially all of the region of interest, the portion of the bodybeing worked on or examined. For non-limiting example, for an abdominaloperation, the field of view would be substantially all of the interiorof the abdomen.

The wide-angle lens can be selected from a group consisting of: afish-eye lens, an omnidirectional lens, any other conventionalwide-angle lens and any combination thereof.

In some embodiments, the display provides a 3D view of the region ofinterest. In preferred embodiments, structured light is used to providethe 3D view.

The structured slight method produces 3D images using a single 2Dcamera. In the structured light method, the object is illuminated by aset of rays of light, each ray illuminating a spot on the object from aknown position and a known direction, and each ray emitted at a knowntime. For each known time, a 2D camera image is created from lightreflected from the spots created from rays existing at that time.Initially, a known calibration object is illuminated. From the knownshape, size and position of the calibration object and from thelocations in the camera images of the reflected light, mathematicalmatrices can be calculated. These matrices enable calculation of the 3Dlocation of the surface of an unknown object, when the unknown object isilluminated by the same set of rays as illuminated the calibrationobject.

In preferred embodiments, the system comprises software for fog removal.Any fog removal technique known in the art can be used. Typical fogremoval algorithms comprise, but are not limited to, adjustment ofbrightness and contrast to compensate for the fog; estimating the fogdensity pixel by pixel and removing it; estimating an overall fogdensity and removing the overall fog density from each pixel; estimatingthe fog density at the deepest point in the image, scaling the fogdensity by the estimated distance to the object, and removing the scaleddensity from the pixel, and any combination thereof.

EXAMPLES

Examples are given in order to prove the embodiments claimed in thepresent invention. The example, which is a clinical test, describes themanner and process of the present invention and set forth the best modecontemplated by the inventors for carrying out the invention, but arenot to be construed as limiting the invention.

In the examples below, similar numbers refer to similar parts in all ofthe figures.

Example 1—Tracking System with Collision Avoidance System

One embodiment of such a rule-based system will comprise the followingset of commands:

Detection (denoted by Gd):Gd₁ Tool location detection functionGd₂ Organ (e.g. Liver) detection functionGd₃ Movement (vector) calculation and estimation functionGd₄ Collision probability detection function

Tool Instructions (denoted Gt):

Gt₁ Move according to manual commandGt₂ Stop movement

The scenario—manual move command by the operator:

Locations Gd₁(t) and Gd₂(t) are calculated in real time at each timestep (from an image or location marker).

Tool movement vector Gd₃(t) is calculated from Gd₁(t) as the differencebetween the current location and at least one previous location(probably also taking into account previous movement vectors).

The probability of collision—Gd₄(t)—is calculated, for example, from thedifference between location Gd₁ and location Gd₂ (the smaller thedistance, the closer the proximity and the higher the probability ofcollision), from movement vector Gd₃(t) indicating a collision, etc.

Tool Instructions Gt ₁ Weight function α₁(t)=1 if Gt ₁(t)<apredetermined threshold and 0 otherwise

Tool Instructions Gt ₂ Weight function α₂(t)=1 if Gt ₂(t)>apredetermined threshold and 0 otherwise

Tool Instructions=α₁(t)*Gt ₁+α₂(t)*Gt ₂(t);

In reference to FIG. 1, which shows, in a non-limiting manner, anembodiment of a tracking system and collision avoidance system. Thesystem tracks a tool 310 and the liver 320, in order to determinewhether a collision between the tool 310 and the liver 320 is possiblewithin the next time step. FIGS. 1a and 1b show how the behavior of thesystem depends on the distance 330 between the tool 310 and the liver320, while FIGS. 1c and 1d show how movement of the tool 310 affects thebehavior. In FIG. 1a , the distance 330 between the tool 310 and theliver 320 is large enough that a collision is not possible in that timestep. Since no collision is possible, no movement of the tool iscommanded. In FIG. 1b , the distance 330 between the tool 310 and theliver 320 is small enough that a collision is likely. In the embodimentillustrated, a movement 340 is commanded to move the tool 310 away fromthe liver 320. In other embodiments, the system prevents movement 350,but does not command movement 340; in such embodiments, the tool 310will remain close to the liver 320. In yet other embodiments, the systemwarns/signals the operator that the move is restricted, but does notrestrict movement 350 or command movement 340 away from the liver. Sucha warning/signaling can be visual or aural, using any of the methodsknown in the art.

FIGS. 1c and 1d illustrate schematically the effect of the movement oftool 310 on the collision avoidance system. In FIGS. 1c and 1d , thetool 310 is close enough to the liver 320 that a collision between thetwo is possible. If the system tracked only the positions of the tool310 and the liver 320, then motion of the tool 310 away from the liver320 would be commanded. FIG. 1c illustrates the effect of a movement 350that would increase the distance between tool 310 and liver 320. Sincethe movement 350 is away from liver 320, no collision is possible inthis time step and no movement of the tool 310 is commanded.

In FIG. 1d , tool 310 is the same distance from liver 320 as in FIG. 1c. However, in FIG. 1d , the movement 350 of the tool 310 is toward theliver 320, making a collision between tool 310 and liver 320 possible.In some embodiments, a movement 340 is commanded to move the tool 310away from the liver 320. In other embodiments, the system preventsmovement 350, but does not command movement 340; in this embodiment thetool 310 will remain close to the liver 320. In yet other embodiments,the system warns the operator that move is restricted, but does notrestrict movement 350 or command movement 340 away from the liver. Sucha warning can be visual or aural, using any of the methods known in theart.

As a non-limiting example, in an operation on the liver, the collisiondetection function can warn the operator that a collision between a tooland the liver is likely but not prevent the collision. In an operationon the gall bladder, the collision detection function can prevent acollision between the tool and the liver, either by preventing themovement or by commanding a movement redirecting the tool away from theliver,

Example 2—Tracking System with Soft Control—Fast Movement when Nothingis Nearby, Slow Movement when Something is Close

One embodiment of such rule-based system comprises the following set ofcommands:

Detection (Denoted by Gd):

Main Tool location detection function (denoted by GdM);Gd-tool₁-K—Tool location detection function;Gd-organ₂-L—Organ (e.g. Liver) detection function;Gd₃ Main Tool Movement (vector) calculation and estimation function;Gd₄ Proximity probability detection function;

Tool Instructions (Denoted Gt):

Gt₁ Movement vector (direction and speed) according to manual command

The scenario—manual move command by the operator:

Locations GdM(t), Gd-tool₁-K(t) and Gd-organ₂-L(t) are calculated inreal time at each time step (from image or location marker).

Main Tool Movement Vector Gd₃(t) is calculated per GdM (t) as thedifference between the current location and at least one previouslocation (probably also taking into account previous movement vectors)

The proximity of the main tool to other tools—Gd₄(t)—is calculated, forexample, as the smallest of the differences between the main toollocation and the other tools' locations.

Tool Instructions Gt₁ Weight function α₁(t) is proportional to toolproximity function Gd₄(t), the closer the tool the slower the movementso that, for example

α₂(t)=Gd ₄/maximum(Gd ₄)

or

α₂(t)=log(Gd ₄/maximum(Gd ₄))

where maximum(Gd₄) is the maximum distance which is likely to result ina collision given the distances, the speed of the tool and the movementvector.

Tool Instructions=α₁(t)*Gt ₁.

Example 3—Tracking System with No-Fly Rule/Function

In reference to FIG. 2, which shows, in a non-limiting manner, anembodiment of a tracking system with no-fly rule. The system tracks atool 310 with respect to a no-fly zone (460), in order to determinewhether the tool will enter the no-fly zone (460) within the next timestep. In this example, the no-fly zone 460 surrounds the liver.

FIGS. 2a and 2b show how the behavior of the system depends on thelocation of the tool tip with respect to the no-fly zone, while FIGS. 2cand 2d show how movement of the tool affects the behavior.

In FIG. 2a , the tool 310 is outside the no-fly zone rule/function 460and no movement of the tool is commanded. In FIG. 2b , the tool 310 isinside the no-fly zone 460.

The no-fly zone rule/function performs as follows:

In the embodiment illustrated, a movement 350 is commanded to move thetool 310 away from the no-fly zone 460. In other embodiments, the systemprevents movement further into the no-fly zone (refers as movement 340,see FIG. 2c ), but does not command movement 340; in such embodiments,the tool 310 will remain close to the no-fly zone 460.

In yet other embodiments, the system warns/signals the operator that themove is restricted, but does not restrict movement further into theno-fly zone or command movement 340 away from the no-fly zone 460. Sucha warning/signaling can be visual or aural, using any of the methodsknown in the art.

FIGS. 2c and 2d illustrate schematically the effect of the tool'smovement on operation of the no-fly zone rule/function. In FIGS. 2c and2d , the tool 310 is close enough to the no-fly zone 460 (distance 330is small enough) that it is possible for the tool to enter the no-flyzone during the next time step. FIG. 2c illustrates the effect of amovement 340 that would increase the distance between tool 310 andno-fly zone 460. Since the movement 340 is away from no-fly zone 460, nocollision is possible in this time step and no movement of the tool 310is commanded.

In FIG. 2d , tool 310 is the same distance from no-fly zone 460 as inFIG. 2c . However, in FIG. 2d , the movement 340 of the tool is towardno-fly zone 460, making it possible for tool 310 to enter no-fly zone460. In the embodiment illustrated, a movement 350 is commanded to movethe tool 310 away from the no-fly zone 460. In other embodiments, thesystem prevents movement 340, but does not command movement 350; in suchembodiments, the tool 310 will remain close to the no-fly zone 460. Inyet other embodiments, the system warns/signals the operator that themove is restricted, but does not restrict movement 340 or commandmovement 350 away from the no-fly zone rule/function 460. Such awarning/signaling can be visual or aural, using any of the methods knownin the art.

Example 4—Tracking System with Preferred Volume Zone Rule/Function

In reference to FIG. 3, which shows, in a non-limiting manner, anembodiment of a tracking system with a preferred volume zonefunction/rule.

The system tracks a tool 310 with respect to a preferred volume zone(570), in order to determine whether the tool will leave the preferredvolume (570) within the next time step.

In this example, the preferred volume zone 570 extends over the rightlobe of the liver. FIGS. 3a and 3b show how the behavior of the systemdepends on the location of the tool tip with respect to the preferredvolume zone 570, while FIGS. 3c and 3d show how movement of the toolaffects the behavior (i.e., the preferred volume zone rule/function).

In FIG. 3a , the tool 310 is inside the preferred volume zone 570 and nomovement of the tool is commanded. In FIG. 3b , the tool 310 is outsidethe preferred volume zone 570.

In the embodiment illustrated, a movement 340 is commanded to move thetool 310 away from the preferred volume zone 570. In other embodiments,the system prevents movement 340; in such embodiments, the tool 310 willremain close to the preferred volume zone 570. In yet other embodiments,the system warns/signals the operator that the move 340 is restricted.Such a warning/signaling can be visual or aural, using any of themethods known in the art.

FIGS. 3c and 3d illustrate schematically the effect of the tool'smovement on operation of the preferred volume rule/function. In FIGS. 3cand 3d , the tool 310 is close enough to the edge of preferred volumezone 570 that it is possible for the tool to leave the preferred volumezone during the next time step.

FIG. 3c illustrates the effect of a movement 350 that would take thetool 310 deeper into preferred volume zone 570. Since the movement 350is into preferred volume 570, the movement is an allowed movement.

In FIG. 3d , the movement 350 of the tool is out of the preferred volume570, making it possible for tool 310 to leave preferred volume 570.

According to one embodiment illustrated, a movement 340 is commanded tomove the tool 310 into the preferred volume zone 570. In otherembodiments, the system prevents movement 350, but does not commandmovement 340; in such embodiments, the tool 310 will remain close to thepreferred volume zone 570. In yet other embodiments, the systemwarns/signals the operator that the move is restricted, but does notrestrict movement 350 or command movement 340 away from the preferredvolume zone 570. Such a warning/signaling can be visual or aural, usingany of the methods known in the art.

Example 5—Organ/Tool Detection Function

In reference to FIG. 4, which shows, in a non-limiting manner, anembodiment of an organ detection system (however, it should be notedthat the same is provided for detection of tools, instead of organs).

For each organ, the 3D spatial positions of the organs stored in adatabase. In FIG. 4, the perimeter of each organ is marked, to indicatethe edge of the volume of 3D spatial locations stored in the database.

In FIG. 4, the liver 610 is labeled with a dashed line. The stomach 620is labeled with a long-dashed line, the intestine 630 with a solid lineand the gall bladder 640 is labeled with a dotted line.

In some embodiments, a label or tag visible to the operator is alsopresented. Any method of displaying identifying markers known in the artcan be used. For non-limiting example, in an enhanced display, coloredor patterned markers can indicate the locations of the organs, with themarker either indicating the perimeter of the organ or the area of thedisplay in which it appears.

Example 6—Tool Detection Function

In reference to FIG. 5, which shows, in a non-limiting manner, anembodiment of a tool detection function. For each tool, the 3D spatialpositions of the tools stored in a database. In FIG. 5, the perimeter ofeach tool is marked, to indicate the edge of the volume of 3D spatiallocations stored in the database. In FIG. 5, the left tool is labeledwith a dashed line while the right tool is labeled with a dotted line.

In some embodiments, a label or tag visible to the operator is alsopresented. Any method of displaying identifying markers known in the artcan be used. For non-limiting example, in an enhanced display, coloredor patterned markers can indicate the locations of the tools, with themarker either indicating the perimeter of the tool or the area of thedisplay in which it appears.

Example 7—Movement Detection Function/Rule

In reference to FIG. 6, which shows, in a non-limiting manner, anembodiment of a movement detection function/rule. FIG. 6a schematicallyillustrates a liver 810, a left tool 820 and a right tool 830 at a timet. FIG. 6b schematically illustrates the liver 810, left tool 820 andright tool 830 at a later time t+Δt, where Δt is a small time interval.In this example, the left tool 820 has moved downward (towards thedirection of liver 810) in the time interval Δt.

The system has detected movement of left tool 820 and labels it. This isillustrated schematically in FIG. 6b by a dashed line around left tool820.

Example 8—Prediction Function

In reference to FIG. 7, which shows, in a non-limiting manner, anembodiment of the above discussed prediction function.

FIG. 7a shows a left tool 920 and a right tool 930 at a time t.

FIG. 7b shows the same tools at a later time t+Δt, where Δt is a smalltime interval. Left tool 920 is moving to the right and downward, whileright tool 930 is moving to the left and upward. If the motion continues(shown by the dashed line in FIG. 7c ), then by the end of the next timeinterval, in other words, at some time between time t+Δt and time t+2Δt,the tools will collide, as shown by tool tips within the dotted circle950 in FIG. 7 c.

In this embodiment, the system automatically prevents predictedcollisions and, in this example, the system applies a motion 940 toredirect left tool 920 so as to prevent the collision.

In other embodiments, the system warns/signals the operator that acollision is likely to occur, but does not alter the movement of anytool. Such a warning/signaling can be visual or aural, using any of themethods known in the art.

In other embodiments, the prediction function can be enabled to, fornon-limiting example, alter the field of view to follow the predictedmovement of a tool or of an organ, to warn of (or prevent) predictedmotion into a no-fly zone, to warn of (or prevent) predicted motion outof a preferred zone.

Example 9—Right Tool Function/Rule

In reference to FIG. 8, which shows, in a non-limiting manner, anembodiment of a right tool function. FIG. 8 schematically illustrates aliver 1010, a left tool 1020 and a right tool 1030. The right tool,illustrated schematically by the dashed line 1040, is labeled and its 3Dspatial location is constantly and real-time stored in a database. Now,according to the right tool function/rule the endoscope constantlytracks the right tool.

It should be pointed out that the same rule/function applies for theleft tool (the left tool function/rule).

It should be further pointed out that paradigm of tracking a tool in aparticular region of the field of view can be extended to any number oftools, for non-limiting example, upper tool function, lower toolfunction, second-from-right function, and second-from-left function.Other such rules/functions will be obvious to one skilled in the art.

Example 10—Field of View Function/Rule

In reference to FIG. 9, which shows, in a non-limiting manner, anembodiment of a field of view function/rule.

FIG. 9a schematically illustrates a field of view of the abdomen at atime t. In the field of view are the liver 1110, stomach 1120,intestines 1130 and gall bladder 1140.

The gall bladder is nearly completely visible at the left of the fieldof view. Two tools are also in the field of view, with their tips inproximity with the liver. These are left tool 1150 and right tool 1160.In this example, the field of view function/rule tracks left tool 1150.In this example, left tool 1150 is moving to the right, as indicated byarrow 1170.

FIG. 9b shows the field of view at time t+Δt. The field of view hasmoved to the right so that the tip of left tool 1150 is still nearly atthe center of the field of view. It can be seen that much less of gallbladder 1140 is visible, while more of right tool 1160 has entered thefield of view.

The field of view function/rule can be set to follow a selected tool, asin this example, or to keep a selected organ in the center of the fieldof view. It can also be set to keep a particular set of tools in thefield of view, zooming in or out as necessary to prevent any of thechosen tools from being outside the field of view.

Alternatively, the field of view function/rule defines n 3D spatialpositions; n is an integer greater than or equal to 2; the combinationof all of the n 3D spatial positions provides a predetermined field ofview.

Each movement of the endoscope or the surgical tool within the n 3Dspatial positions is an allowed movement and any movement of theendoscope or the surgical tool outside the n 3D spatial positions is arestricted movement.

Alternatively, the field of view function/rule defines n 3D spatialpositions; n is an integer greater than or equal to 2; the combinationof all of the n 3D spatial positions provides a predetermined field ofview.

According to the field of view function/rule, the endoscope is relocatedif movement has been detected by the detection means, such that thefield of view is maintained.

Example 11—Tagged Tool Function/Rule (or Alternatively the PreferredTool Rule)

In reference to FIG. 10, which shows, in a non-limiting manner, anembodiment of a tagged tool function/rule.

FIG. 10 shows three tools (1220, 1230 and 1240) in proximity to theorgan of interest, in this example, the liver 1210.

The tool most of interest to the operator, at this point during theoperation, is tool 1240. Tool 1240 has been tagged (dotted line 1250);the 3D spatial location of tool 1240 is constantly stored in a databaseand this spatial location has been labeled as one of interest.

The system can use this tagging for many purposes, including, but notlimited to, keeping tool 1240 in the center of the field of view,predicting its future motion, keeping it from colliding with other toolsor keeping other tools from colliding with it, instructing the endoscopeto constantly monitor and track the tagged tool 1250 and so on.

It should be noted that in the preferred tool rule, the system tags oneof the tools and performs as in the tagged tool rule/function.

Example 12—Proximity Function/Rule

In reference to FIG. 11, which shows, in a non-limiting manner, anembodiment of a proximity function/rule.

FIG. 11a schematically illustrates two tools (1310 and 1320) separatedby a distance 1330 which is greater than a predefined proximitydistance. Since tool 1310 is not within proximity of tool 1320, thefield of view (1380) does not move.

FIG. 11b schematically illustrates two tools (1310 and 1320) separatedby a distance 1330 which is less than a predefined proximity distance.

Since tool 1310 is within proximity of tool 1320, the field of view 1380moves upward, illustrated schematically by arrow 1340, until the tips oftool 1310 and tool 1320 are in the center of field of view 1380 (FIG.11c ).

Alternatively the once the distance 1330 between the two tool 1320 and1310 is smaller than a predetermined distance, the system alerts theuser of the proximity (which might lead to a collision between the twotools). Alternatively, the system moves one of the tools away from theother one.

Example 13—Operator Input Function/Rule

In reference to FIG. 12, which shows, in a non-limiting manner, anembodiment of an operator input function/rule. According to thisembodiment, input is received from the operator.

In the following example, the input received from the operator is whichtool to track.

FIG. 12a schematically illustrates an endoscope with field of view 1480showing a liver 1410 and two tools 1420 and 1430. A wireless transmitter1460 is enabled to transmit coded instructions through receiver 1470.Operator 1450 first selects the tip of the left tool as the region ofinterest, causing the system to tag (1440) the tip of the left tool.

As illustrated in FIG. 12b , the system then directs and modifies thespatial position of the endoscope so that the tagged tool tip 1440 is inthe center of the field of view 1480.

Another example of the operator input function/rule is the following:

If a tool has been moved close to an organ in the surgical environment,according to the proximity rule or the collision prevention rule, thesystem will, according to one embodiment, prevent the movement of thesurgical tool.

According to one embodiment of the present invention, once the surgicaltool has been stopped, any movement of the tool in a direction towardthe organ is interpreted as input from the operator to continue themovement of the surgical tool in that direction.

Thus, according to this embodiment, the operator input function/rulereceives input from the operator (i.e., physician) to continue to moveof the surgical tool, even though it violates the collision preventionrule. The input is simply in the form of the continued movement of thesurgical tool after the alert of the system or after movement preventionby the system.

Example 14—Constant Field of View Rule/Function

In reference to FIGS. 13A-D, which shows, in a non-limiting manner, anembodiment of a tracking system with a constant field of viewrule/function.

In many endoscopic systems, the tip lens in the camera optics is not ata right angle to the sides of the endoscope. Conventionally, the tiplens angle is described relative to the right angle, so that a tip lensat right angles to the sides of the endoscope is described as having anangle of 0. Typically, angled endoscope tip lenses have an angle of 30°or 45°. This tip lens angle affects the image seen during zooming. FIG.13A-D illustrates, in an out-of-scale manner, for a conventional system,the effect of zooming in the field of view in an endoscope with tip lensset straight in the end (FIGS. 13A and 13B) vs. the effect of zooming inthe field of view in an endoscope with angled tip lens (FIGS. 13C and13D).

FIGS. 13A and 13C illustrate the endoscope (100), the object it isviewing (200) and the image seen by the endoscope camera (130) beforethe zoom. The solid arrows (160) show the limits of the FOV and thedashed arrow (170), the center of the field of view (FOV); since theobject is in the center of the FOV, an image of the object (210) is inthe center of the camera image (130). FIGS. 13B and 13D illustrate theendoscope (100), the object it is viewing (200) and the image seen bythe endoscope camera (130) after the zoom. The solid arrows (160) showthe limits of the FOV and the dashed arrow (170), the center of thefield of view.

If the tip lens is set straight in the end of the endoscope (FIGS. 13Aand 13B), an object (200) in the center of the field of view will be inthe center of the field of view (FOV) (and the camera image) (130) bothbefore (FIG. 13A) and after (FIG. 13B) the zoom. However, if the tiplens is set at an angle in the end of the endoscope (FIGS. 13C and 13D),then an object that is in the center of the FOV (and the camera image)before the zoom (FIG. 13C) will not be in the center of the FOV (or thecamera image) after the zoom (FIG. 13D) since the direction of motion ofthe endoscope is not the direction in which the center of the field ofview (170) points.

In an embodiment of the system of the present invention, unlike inconventional systems, the controlling means maintains the center of thefield of view (FOV) during zoom independent of the tip lens angle. Anadvantage of controlling the zoom of the endoscope via a data processingsystem is that the tip lens angle does not need to be input to the dataprocessing system, obviating a possible source of error.

According to one embodiment of the present invention, the endoscope'smovement will be adjusted in order to maintain a constant field of view.

Example 15—Misalignment Rule/Function

According to another embodiment of the present invention, the system caninform the user of any misalignment of the same system.

Misalignment of the system may cause parasitic movement of the endoscopetip, where the endoscope tip does not move exactly in the expecteddirection. According to one embodiment of the system, the systemcomprises sensors (e.g., gyroscopes, accelometers and any combinationthereof) that calculate/estimates the position of the pivot point inreal time in order to (a) inform the user of misalignment; or (b)calculate the misalignment so that the system can adjust its movement toprevent parasitic movement.

Example 16—Change of Speed Rule/Function

In reference to FIG. 14, which shows, in a non-limiting manner, anembodiment of a tracking system with a change of speed rule/function.

In conventional endoscopic control systems, motion of the endoscopeoccurs at a single speed. This speed is fairly fast so that theendoscope can be moved rapidly between locations that are wellseparated. However, this means that making fine adjustments so difficultthat fine adjustments are normally not made. In an embodiment of thepresent invention, the speed of the tip of the endoscope isautomatically varied such that, the closer the endoscope tip is to anobject, be it a tool, an obstacle, or the object of interest, the moreslowly it moves.

In this embodiment, as shown in FIG. 14, measurements are made of thedistance X (150) from the tip (195) of the endoscope (100) to the pivotpoint of the endoscope (190), where the pivot point is at or near thesurface of the skin (1100) of a patient (1000). Measurements are alsomade of the distance Y (250) from the tip of the endoscope (195) to theobject in the center of the scene of view (200). From a predeterminedvelocity V_(p), the actual velocity of the tip of the endoscope at agiven time, V_(act), is calculated from

$V_{act} \propto {\frac{Y}{X}V_{p}}$

Therefore, the closer to the object at the center of the scene of view,the more slowly the endoscope moves, making it possible to use automaticcontrol of even fine adjustments, and reducing the probability that theendoscope will come in contact with tissue or instruments.

Example 17—Input Movement Protocol, Movement of a Tool

Non-limiting examples of input movement protocols involving movement oftools and associated output movement protocols will be given. Forsimplicity, the input commands comprise a single movement protocol. Itis clear that a movement command can comprise any number of movementprotocols, positions, repositions and actions,

In reference to FIG. 15A-B, which shows, in a non-limiting manner, anembodiment of an input movement protocol comprising shaking a tool.

In FIG. 15A, a system comprising three tools (1520, 1530, 1540) isillustrated; the system is tracking (dashed line) the upper right tool(1530). In order to change tracking to the leftmost tool (1520), theleftmost tool (1520) is shaken (1550, dotted line)

As shown in FIG. 15B, once the leftmost tool (1520) has been shaken.according to the output movement protocol, the system tracks (dashedline) the leftmost tool (1520).

In this example, for clarity, a single tool is tracked. It is clear thata plurality of tools can be simultaneously tracked.

In reference to FIG. 16A-B, which shows, in a non-limiting manner, anembodiment of a zoom command.

In FIG. 16A, two tools (1620, 1630) are being used in an operation onthe liver (1610). To command a zoom inward, the tip of a tool, in thiscase, the right tool (1630), is moved in a clockwise circle (1650,dotted line).

As shown in FIG. 16B, once the circle has been made, according to theoutput protocol, the field of view is zoomed inward, keeping the centerof the field of view the same, so that the image is magnified by 50%.

In this embodiment, an input protocol of a counterclockwise circle (notshown) of either tool would result in an output movement protocol of azoom outward, increasing the field of view and demagnifying the image by50%.

The embodiments shown herein are merely exemplary—there are many inputmovement protocols and many output movement protocols which have notbeen shown.

It should be noted that the association of input and output movementprotocols is arbitrary; any input movement protocol can be associatedwith any output protocol.

Example 18—Input Movement Protocol, Movement of an Operator

Non-limiting examples of input movement protocols involving movement ofa part of an operator, in this case the hand, and associated outputmovement protocols will be given. For simplicity, each input movementcommand comprises a single movement protocol, a predetermined gesture.It is clear that a movement command can comprise any number of movementprotocols, as well as positions, repositions and actions.

In reference to FIG. 17A-C, which shows, in a non-limiting manner, anembodiment of an input movement protocol comprising pointing a finger.

In FIG. 15A, a system comprising three tools (1520, 1530, 1540) isillustrated; the system is tracking (dashed line) the upper right tool(1530). As shown in FIG. 17B, in order to change tracking to theleftmost tool (1520), the operator points to the left (1750), in thiscase with the right hand.

As shown in FIG. 17C, once operator has pointed, according to the outputmovement protocol, the system tracks (dashed line) the leftmost tool(1520).

In this example, for clarity, a single tool is tracked. It is clear thata plurality of tools can be simultaneously tracked.

In reference to FIG. 18A-C, which shows, in a non-limiting manner, anembodiment of an input movement protocol for centering a field of view.

In this embodiment, the input movement protocol to place the center ofthe field of view at the tip of the tracked tool is holding the handopen downward with the finger spread as though picking up a bowl (FIG.18A, 1850).

As shown in FIG. 18B, the tip of the tracked tool (1880, dashed line) isto the left of the center of the field of view, which shows two tools(1880, 1890), the liver (1810) and the stomach (1820).

The gesture (FIG. 18A, 1850) commands the output movement protocol, thatthe center of the field of view be moved to the right (dashed arrow,1870). After the output movement protocol has been completed, the tip ofthe tracked, left, tool (1880, dashed line) is at the center of thefield of view, which shows the two tools (1880, 1890), liver (1810), thestomach (1820), the intestines (1830) and gall bladder (1840).

In this example, for clarity, the center of the field of view follows asingle tool. It is clear that the center of the field of view can dependon the locations of a plurality of tools.

In reference to FIG. 19A-C, which shows, in a non-limiting manner, anembodiment of an input movement protocol to zoom an endoscope.

In this embodiment, the input movement protocol to zoom the endoscopeinward is holding an open hand sideways with the fingers together,although picking up a book (FIG. 19A, 1950).

In FIG. 19B, two tools (1920, 1930) are being used in an operation onthe liver (1910).

As shown in FIG. 19C, once the input protocol (holding the hand asthough picking up a book) is made, according to the output protocol, thefield of view is zoomed inward, keeping the center of the field of viewthe same, so that the image is magnified by 50%.

In this embodiment, an input movement protocol of a book-holding gesturepointing toward the right would result in an output movement protocol ofa zoom outward, increasing the field of view and demagnifying the imageby 50%.

The embodiments shown herein are merely exemplary—there are many inputmovement protocols and many output movement protocols which have notbeen shown.

It should be noted that the association of input and output movementprotocols is arbitrary; any input movement protocol can be associatedwith any output protocol.

Example 19—Input Movement Protocol, Movement of an Operator

A non-limiting example of an input movement protocol comprising movementof a part of an operator, in this case the eye, and an associated outputmovement protocol will be given. For simplicity, the input movementprotocol comprises a single fixed predetermined gesture. It is clearthat a movement command can comprise any number of movement protocols,as well as positions, repositions and actions.

In reference to FIG. 20A-C, which shows, in a non-limiting manner, anembodiment of an input movement protocol comprising moving at least oneeye.

In FIG. 20A, a system comprising three tools (2020, 2030, 2040) isillustrated; the system is tracking (dashed line) the upper right tool(2030). In order to change tracking to the leftmost tool (2020), atleast one eye is moved to look upward to the left, preferably so thatthe operator is no longer looking at the display screen, as shown inFIG. 20B (2030). In preferred embodiments, the eye gesture need only bea quick glance, a momentary removal of the eyes from the display.

As shown in FIG. 20C, once the eye gesture (2060) is complete, accordingto the output movement protocol, the system tracks (dashed line) theleftmost tool (2020).

In this example, for clarity, a single tool is tracked. It is clear thata plurality of tools can be simultaneously tracked

Example 20—Input Protocol, Position of a Tool

A non-limiting example of an input movement command comprising aposition of a tool is shown in FIG. 21A-B. The input movement commandcomprises an input position. In other embodiments, the input movementcommand can comprise a plurality of positions, repositions, actions andmovement protocols.

In FIG. 21A, an embodiment of a display image is shown. The displaycomprises at least one icon (2150), with each icon being associated withan output command. In this embodiment, icons are invisible until a tool“enters” an icon, in other words, until the image of the tool is in theregion of the display which can show the icon. In other embodiments, atleast some icons are visible at all times.

In this embodiment, once a tool (2130) has entered an icon (2150), theoutput command is activated by moving the tool in a gesture whichencircles the icon (2160, dotted arrow). In other embodiments, enteringthe icon region activates the output protocol; in yet other embodiments,other gestures are used.

In this exemplary embodiment, the icon (2150) shows a zoom-inward (+)symbol. After the circling motion (2160, dotted arrow) is completed, thesystem zooms the endoscope inward until the tool is removed from theicon, whereupon zooming stops and a magnified image is shown (FIG. 21B).The location of the icon is shown greyed-out in FIG. 21B forillustrative purposes. In preferred variants of this embodiment, an iconwould only be showed greyed-out if the function with which it isassociated is unavailable. In preferred variants, icons are preferablyoutside the image of the field of view or invisible when not in use, inorder to ensure that the image of the field of view is as visible aspossible.

In this example, for clarity, a single tool is shown. It is clear thatany of a plurality of tools can be positioned over the icon.

Example 21—Input Protocol, Tagging of an Object

A non-limiting example of an input command comprising an action by amoving element is shown in FIG. 22A-B. For simplicity, the input commandcomprises a single action. In other embodiments, the input command cancomprise a plurality of positions, repositions, actions and movementprotocols.

In this embodiment, as shown in FIG. 22A, the command is pointing by afinger of an operator (2250) at the object (2260) to be tagged.

As shown in FIG. 22B, the output protocol tags (2260, dashed line) theobject, centers it in the field of view, and zooms the object until itis entirely within the field of view and fills the field of view in atleast one direction.

In this example, for clarity, a single tagged object is used. It isclear that any of a plurality of tagged objects can be kept within thefield of view.

Example 22—Input Protocol, Activation of a Tool

A non-limiting example of an input command comprising an action ofactivating a tool is shown in FIG. 23A-B. For simplicity, the inputcommand comprises a single action; in other embodiments, the inputcommand can comprise a plurality of positions, repositions, actions andmovement protocols.

In this embodiment, as shown in FIG. 23A, the tool (2330) is a grasperand activation comprises closing the grasper (2350, curved arrows).

Closing (2350, curved arrows) of the grasper (2330) results in an outputprotocol in which (FIG. 23B) the tip (2335, dashed circle) of thegrasper (2330) is placed in the center of the field of view and the viewzoomed to give a good view of the tip of the grasper.

In this example, for clarity, a single tool is activated. It is clearthat any of a plurality of tools can be activated, and that theactivated tools need not be of the same type (e.g., a cautery and agraspers).

Example 23—Input Protocol, Tool Reaches Edge of Field of View

A non-limiting example of an input command to keep a tagged object fromreaching an edge of the field of view is shown in FIG. 24A-B.

In this embodiment, as shown in FIG. 24A, the tagged object is a tool(2420). Location of the tip of the tool anywhere in the area between apredetermined distance (2450, dotted line) and the edge of the field ofview determines activation of the input command that the tool tip is tobe kept within the field of view. This, in turn, activates an outputcommand to maneuver the endoscope so as to place the tip (2425) of thetool (2420) in the center of the field of view, as is shown in FIG. 24B.

In other embodiments, more than one article can be kept from the edge ofthe field of view. In such embodiments, a plurality of articles can betagged. If a tagged article reaches an edge of the field of view, theendoscope will maneuver to move the article away from the edge. In somevariants of these embodiments, in addition to, or in place of,maneuvering the endoscope, the endoscope's zoom will be altered untilall the tagged articles are more than the predetermined distance fromthe edge.

Example 24—Relationship Between Articles

A non-limiting example of a relationship between articles is shown inFIG. 25.

In this example, a fluid delivery tube (2520) and a suction tube (2530)are kept at fixed distances (2540, 2550), which are not the same, from acautery (2510). A predetermined angle (2560) is maintained between thetip of the fluid delivery tube (2520), the tip of the cautery (2510) andthe tip of the suction tube (2530). In addition, the longitudinal axesof the tubes are at fixed angles (2570, 2580), not the same, relative tothe longitudinal axis of the cautery.

The embodiments shown hereinabove are merely exemplary—there are manyinput movement protocols, many output movement protocols and manyassociations between input command and output command which are possibleand have not been shown.

It should be noted that the association of input and output commands aretypically arbitrary; any input command can be associated with any outputcommand.

It should further be noted that an input command can comprise any of atool movement, an operator movement and an operator brain signal, andthat these can be combined in any way.

In preferred embodiments, the input commands will be chosen so as tomake the system operate as intuitively as is practicable.

In the foregoing description, embodiments of the invention, includingpreferred embodiments, have been presented for the purpose ofillustration and description. They are not intended to be exhaustive orto limit the invention to the precise form disclosed. Obviousmodifications or variations are possible in light of the aboveteachings. The embodiments were chosen and described to provide the bestillustration of the principals of the invention and its practicalapplication, and to enable one of ordinary skill in the art to utilizethe invention in various embodiments and with various modifications asare suited to the particular use contemplated. All such modificationsand variations are within the scope of the invention as determined bythe appended claims when interpreted in accordance with the breadth theyare fairly, legally, and equitably entitled.

1. A method of controlling a surgical tool, comprising: positioning anendoscope, a first surgical tool and a second surgical tool within abody cavity in which a surgical environment is located; capturing, withthe endoscope, real-time images of at least the first and secondsurgical tools in a field of view within the body cavity; real-timeimage processing said images to detect movement of at least a portion ofsaid first surgical tool; determining if said detected movement iswithin one of the plurality of predetermined protocols of inputmovement; if said detected movement is within a predetermined protocolof input movement, activating a corresponding predetermined outputcommands such that the second surgical tool is at least one of thefollowing: robotically maneuvered according to said at least one outputprotocol; and activated according to said at least one output protocol.2. The method of claim 1, wherein the second surgical tool isrobotically maneuvered according to said at least one output protocol.3. The method of claim 1, wherein the second surgical tool is activatedaccording to said at least one output protocol.
 4. The method of claim3, wherein activating the second surgical tool comprises opening orclosing an end effector of the tool.
 5. The method of claim 3, whereinactivating the second surgical tool comprises energizing anelectrosurgical tool.
 6. The method of claim 3, wherein activating thesecond surgical tool comprises activating a stapler.
 7. The method ofclaim 3, wherein activating the second surgical tool comprisesactivating a stapler.
 9. The method of claim 3, wherein activating thesecond surgical tool comprises opening or closing the second surgicaltool.
 10. The method of claim 3, wherein activating the second surgicaltool comprises causing the second surgical tool to function or stopfunctioning.
 11. The method of claim 3, wherein activating the secondsurgical tool comprises causing the second surgical tool to beintroduced into the surgical environment or removed from the surgicalenvironment.
 12. The method of claim 1, wherein said protocol of inputmovement is selected from a group consisting of: moving said firstsurgical tool parallel to the X axis; moving said first surgical toolparallel to the Y axis; moving said first one surgical tool parallel tothe Z-axis; rotational movement of said first surgical tool around anaxis parallel to the X axis; rotational movement of said first surgicaltool around an axis parallel to the Y axis; rotational movement of saidfirst surgical tool around an axis parallel to the Z axis; shaking saidfirst surgical tool, moving said first surgical tool in at least aportion of a circle, moving said first surgical tool in at least aportion of an oval, moving said first surgical tool in at least aportion of an ellipse, moving said first surgical tool in a straightline, moving said first surgical tool in a zigzag, moving said firstendoscope parallel to the X axis, and any combination thereof.
 13. Themethod of claim 1, wherein said protocol of input movement is selectedfrom the group consisting of: at least a portion of said first surgicaltool is positioned in a predetermined region of said field of view; atleast a portion of said first surgical tool is positioned less than apredetermined distance from an edge of said field of view; at least aportion of said first surgical tool is oriented at a predetermined anglein said field of view; there exists a predetermined relationship betweenat least two articles in said field of view and any combination thereof.14. The method of claim 1, wherein the method includes roboticallymaneuvering at least one of the endoscope, first surgical tool andsecond surgical tool in least three dimensions.
 15. The method of claim1, wherein the output protocol is an output movement protocol, andwherein a relationship between magnitude of an output movement andmagnitude of said detected movement is selected from a group consistingof: the magnitude of said output movement is proportional to themagnitude of said detected movement; the magnitude of said outputmovement is substantially identical to the magnitude of said detectedmovement; the magnitude of said output movement is independent of themagnitude of said detected movement
 16. The method of claim 1, whereinthe output protocol is selected from an output movement protocol groupconsisting of an allowed output movement protocol, a restricted outputmovement protocol and any combination thereof of a member of amaneuverable object group consisting of said at least one endoscope,said first surgical tool, said second surgical tool and any combinationthereof, wherein each member of said output movement protocol group isdeterminable from input movement protocols comprising historicalmovements of said member of said maneuverable object group accordingwith historical movement patterns of said member of said maneuverableobject group in at least one previous surgery.
 17. The method of claim16, wherein each member of a group consisting of an allowed inputmovement protocol, a restricted input movement protocol and anycombination thereof comprises, stored in said communicable database,each 3D spatial position of said member of said maneuverable objectgroup according with at least two 3D spatial positions of said member ofsaid maneuverable object group, such that each movement pattern of saidmember of said maneuverable object group and each 3D position of saidmember of said maneuverable object group according with the same isstored; in said associated allowed output protocol, said allowedmovements of said member of said maneuverable object group are movementsin which the same is located substantially in at least one of theendoscope 3D spatial positions according with at least one said 3Dmovement pattern, and said restricted movements are movements in whichthe location of said member of said maneuverable object group issubstantially different from the n 3D spatial positions of the sameaccording with the n movement patterns.
 18. The method of claim 16,wherein each member of a group consisting of an allowed output movementprotocol, a restricted output movement protocol and any combinationthereof comprises at least one rule according to which allowed andrestricted movements of said second surgical tool are determined, suchthat each detected movement of said at least one surgical tool isdetermined as either an allowed movement or as a restricted movementaccording to said predetermined set of rules.
 19. The method of claim18, wherein said allowed movement is permitted by said controller andsaid restricted movement is denied by said controller.
 20. A method ofcontrolling a surgical tool, comprising: positioning an endoscope and afirst surgical tool within a body cavity in which a surgical environmentis located; capturing, with the endoscope, real-time images of at leastthe first surgical tool in a field of view within the body cavity;real-time image processing said images to detect at least one inputcommand from said image of said field of view and to, if said inputcommand is within one of a plurality of predetermined input protocolsactivate at least one of a predetermined plurality of output commandswherein the input command is selected from the group consisting of: a.at least a portion of the first surgical tool is positioned in apredetermined region of said field of view; b. at least a portion of thefirst surgical tool is positioned less than a predetermined distancefrom an edge of said field of view; c. at least a portion of the firstsurgical tool is oriented at a predetermined angle in said field ofview; d. the first surgical tool is activated; e. the first surgicaltool is deactivated; and wherein said output command is selected from agroup consisting of: a. at least a portion of the first surgical tool isrepositioned to a predetermined region of said field of view; b. atleast a portion of the first surgical tool is oriented at apredetermined angle in said field of view; d. the first surgical tool isactivated; e. the first surgical tool is deactivated; f. in the firstsurgical tool, at least one of a group consisting of an articulationangle, an articulation length and any combination thereof is altered; g.the first surgical tool is tagged.
 21. The method of claim 20, whereinsaid activation is selected from a group consisting of: opening saidfirst surgical tool, closing said first surgical tool, causing saidfirst surgical tool to function, stopping said first surgical tool fromfunctioning, introducing said first surgical tool to the surgicalenvironment, and removing said first surgical tool from the surgicalenvironment, and any combination thereof.